design-clone 1.1.1 → 2.1.0

package/docs/cli-reference.md

@@ -22,10 +22,52 @@ node src/core/screenshot.js [options]
 | --close | bool | false | Close browser after capture (false keeps session) |
 | --extract-html | bool | false | Extract cleaned HTML |
 | --extract-css | bool | false | Extract all CSS from page |
+| --extract-animations | bool | true* | Extract @keyframes and transitions (enabled with --extract-css) |
 | --filter-unused | bool | true | Filter CSS to remove unused selectors |
+| --capture-hover | bool | false | Capture hover state screenshots and generate :hover CSS |
+| --no-semantic | bool | false | Disable WordPress semantic HTML enhancement (Phase 3) |
 | --verbose | bool | false | Verbose logging |
 
-**Output**: JSON with screenshot paths and metadata. Includes `browserRestarts` count tracking for stability monitoring.
+*Default true when --extract-css is enabled, can be disabled with `--extract-animations false`
+
+**Output**: JSON with screenshot paths and metadata. Includes `browserRestarts` count tracking for stability monitoring. When `--capture-hover` is enabled, also includes hover state results in output. When semantic enhancement is enabled (default), output includes `semanticStats` with enhancement details (sections enhanced, IDs/classes/roles added).
+
+### Semantic HTML Enhancement (Phase 3)
+
+Semantic HTML enhancement is enabled by default when extracting HTML. It injects WordPress-compatible semantic IDs, classes, and ARIA roles into the extracted HTML.
+
+**What's Added**:
+- **IDs**: `site-header`, `main-content`, `site-footer`, `site-navigation`, `primary-sidebar`, `hero-section`
+- **Classes**: `site-header`, `main-navigation`, `nav-menu`, `site-main`, `content-area`, `widget-area`, `sidebar`, `site-footer`, `hero`
+- **ARIA Roles**: `banner` (header), `navigation` (nav), `main`, `complementary` (sidebar), `contentinfo` (footer)
+
+**Detection Priority**:
+1. Semantic HTML tags (`<header>`, `<nav>`, `<main>`, `<aside>`, `<footer>`)
+2. ARIA role attributes (`banner`, `navigation`, `main`, `complementary`, `contentinfo`)
+3. Class patterns (header, nav, main, sidebar, footer, hero)
+
+**Usage**:
+```bash
+# Enable semantic enhancement (default)
+node src/core/screenshot.js --url https://example.com --output ./out --extract-html
+
+# Disable semantic enhancement
+node src/core/screenshot.js --url https://example.com --output ./out --extract-html --no-semantic
+```
+
+**Example Output Metadata**:
+```json
+{
+  "html": "path/to/source.html",
+  "semanticStats": {
+    "sectionsEnhanced": 5,
+    "idsAdded": 3,
+    "classesAdded": 4,
+    "rolesAdded": 2,
+    "warnings": []
+  }
+}
+```
 
 ## filter-css.js
 
@@ -82,9 +124,62 @@ node src/core/extract-assets.js --url URL --output DIR
 Validate navigation structure.
 
 ```bash
-node src/verification/verify-menu.js --html FILE
+node src/verification/verify-menu.js --html FILE [--url URL] [--output DIR] [--verbose]
+```
+
+| Option | Description |
+|--------|-------------|
+| --html | Path to HTML file |
+| --url | URL to test (alternative to --html) |
+| --output | Output directory for screenshots |
+| --verbose | Show detailed progress |
+
+## verify-header.js
+
+Verify header components (Phase 1).
+
+```bash
+node src/verification/verify-header.js --html FILE [--url URL] [--output DIR] [--verbose]
+```
+
+Tests: logo presence, navigation visibility, CTA buttons, sticky/fixed behavior, z-index layering, height consistency.
+
+## verify-footer.js
+
+Verify footer components (Phase 1).
+
+```bash
+node src/verification/verify-footer.js --html FILE [--url URL] [--output DIR] [--verbose]
+```
+
+Tests: position at bottom, multi-column layout, link sections, copyright text, social icons, background contrast.
+
+## verify-slider.js
+
+Verify slider/carousel components (Phase 1).
+
+```bash
+node src/verification/verify-slider.js --html FILE [--url URL] [--output DIR] [--verbose]
 ```
 
+Tests: library detection (Swiper, Slick, Owl, native), navigation arrows, pagination dots, autoplay behavior, current slide indicator.
+
+## generate-audit-report.js
+
+Aggregate verification results into consolidated report (Phase 1).
+
+```bash
+node src/verification/generate-audit-report.js --dir DIR [--output FILE] [--verbose]
+```
+
+| Option | Description |
+|--------|-------------|
+| --dir | Directory containing verification JSON results |
+| --output | Output path for report (default: component-audit.md) |
+| --verbose | Show detailed progress |
+
+Output: Markdown report with summary table, side-by-side screenshots, responsive analysis, CSS suggestions.
+
 ## verify-layout.js
 
 Verify layout consistency.
@@ -92,3 +187,130 @@ Verify layout consistency.
 ```bash
 node src/verification/verify-layout.js --html FILE
 ```
+
+## state-capture.js
+
+Capture hover states for interactive elements.
+
+Used internally by screenshot.js with `--capture-hover` flag.
+
+| Export | Description |
+|--------|-------------|
+| `captureAllHoverStates(page, cssString, outputDir)` | Detect interactive elements and capture normal/hover screenshots |
+| `captureHoverState(page, selector, outputDir, index)` | Capture hover state for a single element |
+| `generateHoverCss(results)` | Generate `:hover` CSS rules from captured style diffs |
+| `detectInteractiveElements(page, cssString)` | Detect interactive elements via CSS analysis and DOM query |
+
+**Key Features:**
+- Dual detection: CSS-based (`:hover` selectors) and DOM-based (interactive elements, transitions)
+- Per-element style diff capture (backgroundColor, color, transform, boxShadow, etc.)
+- Automatic screenshot pair generation (normal + hover states)
+- CSS rule generation from detected style changes
+- Validates selectors and skips hidden/invisible elements
+
+## ux-audit.js
+
+UX quality assessment using Gemini Vision AI (Phase 2).
+
+```bash
+node src/ai/ux-audit.js --screenshots <dir> [--output <dir>] [--url <url>] [--verbose]
+```
+
+| Option | Required | Description |
+|--------|----------|-------------|
+| --screenshots | yes | Directory containing viewport screenshots (desktop.png, tablet.png, mobile.png) |
+| --output | no | Output directory for report and JSON results (default: same as screenshots) |
+| --url | no | Original URL (for report metadata) |
+| --verbose | no | Show detailed progress |
+
+**Requires**: GEMINI_API_KEY or GOOGLE_API_KEY environment variable
+
+**Output**:
+- `ux-audit.md`: Markdown report with scores, issues, and recommendations
+- `ux-audit.json`: Structured results (aggregated scores, viewport breakdown, issues, recommendations)
+
+**Evaluation Categories** (0-100 score each):
+1. Visual Hierarchy - Content prominence, scanning patterns, call-to-action visibility
+2. Navigation - Touch targets, menu discoverability, current page indicator
+3. Typography - Text size, line height, contrast ratio, readability
+4. Spacing - Padding/margins, element breathing room, touch target spacing
+5. Interactive Elements - Button affordance, link distinguishability, focus states
+6. Responsive - Content reflow, no horizontal scroll, text truncation, breakpoint transitions
+
+**Viewport Analysis**: Evaluates all three viewports (desktop: 1920×1080, tablet: 768×1024, mobile: 375×812) and generates weighted scores (desktop 40%, tablet 30%, mobile 30%).
+
+**Issue Severity Levels**:
+- Critical (0-30 score): Blocks tasks or causes confusion
+- Major (31-60 score): Degrades experience significantly
+- Minor (61-80 score): Polish improvements
+
+**Scoring Scale**:
+- 90-100: Excellent, industry-leading UX
+- 70-89: Good, meets modern standards
+- 50-69: Adequate, room for improvement
+- 30-49: Poor, significant issues
+- 0-29: Critical, requires immediate attention
+
+## clone-site.js
+
+Clone multiple pages from website with integrated UX audit (Phase 2).
+
+```bash
+design-clone clone-site <url> [options]
+```
+
+| Option | Description |
+|--------|-------------|
+| --pages <paths> | Comma-separated paths (e.g., /,/about,/contact) |
+| --max-pages <n> | Maximum pages to auto-discover (default: 10) |
+| --viewports <list> | Viewport list (default: desktop,tablet,mobile) |
+| --output <dir> | Custom output directory |
+| --ai | Extract design tokens using Gemini AI (requires GEMINI_API_KEY) |
+| --ux-audit | Run UX audit using Gemini Vision (requires GEMINI_API_KEY) |
+| --yes, -y | Skip confirmation prompt |
+
+**Integrated Workflow** (when using --ux-audit):
+1. Discover or use manual pages
+2. Capture screenshots across viewports
+3. Merge CSS files
+4. Extract design tokens (with --ai)
+5. **Run UX audit** (with --ux-audit) - Analyzes homepage screenshots via Gemini Vision
+6. Rewrite links
+7. Generate manifest
+
+**UX Audit Output**: When enabled, generates `analysis/ux-audit.md` and `analysis/ux-audit.json` in output directory.
+
+**Examples**:
+```bash
+design-clone clone-site https://example.com --ux-audit
+design-clone clone-site https://example.com --pages /,/about,/contact --ux-audit
+design-clone clone-site https://example.com --ai --ux-audit
+```
+
+## ux_audit.py
+
+Python module providing UX audit prompts for Gemini Vision integration.
+
+```python
+from src.ai.prompts.ux_audit import build_ux_audit_prompt, build_aggregation_prompt
+
+# Build viewport-specific prompt
+prompt = build_ux_audit_prompt(viewport='mobile')
+
+# Build aggregation prompt for multiple viewports
+aggregation = build_aggregation_prompt(desktop_results, tablet_results, mobile_results)
+```
+
+**Functions**:
+- `build_ux_audit_prompt(viewport)` - Build prompt with viewport-specific checks (mobile/tablet/desktop)
+- `build_aggregation_prompt(desktop, tablet, mobile)` - Combine viewport results into unified assessment
+
+**Constants**:
+- `UX_AUDIT_PROMPT` - Base UX evaluation prompt (6 categories)
+- `VIEWPORT_CONTEXT` - Dictionary of viewport-specific evaluation criteria
+- `AGGREGATION_PROMPT` - Template for combining viewport results with weighted averaging
+
+**Viewport Weighting**:
+- Desktop: 40% (primary interaction model)
+- Tablet: 30% (hybrid interaction)
+- Mobile: 30% (touch-first)

package/docs/codebase-summary.md

@@ -0,0 +1,309 @@
+# Codebase Summary
+
+## Overview
+
+Design Clone is a comprehensive design extraction toolkit that captures website designs through multi-viewport screenshots, extracts HTML/CSS, analyzes structure with AI, and enhances semantic HTML for WordPress compatibility.
+
+## Core Architecture
+
+### Directory Structure
+
+```
+design-clone/
+├── src/
+│   ├── core/                          # Core extraction & processing modules
+│   │   ├── screenshot.js              # Multi-viewport screenshot capture
+│   │   ├── html-extractor.js          # HTML extraction + semantic enhancement
+│   │   ├── semantic-enhancer.js       # WordPress semantic HTML injection (Phase 3)
+│   │   ├── css-extractor.js           # CSS extraction & property tracking
+│   │   ├── filter-css.js              # Unused CSS selector removal
+│   │   ├── animation-extractor.js     # @keyframes & transition extraction
+│   │   ├── state-capture.js           # Hover state capture
+│   │   ├── extract-assets.js          # Image/font/icon downloading
+│   │   ├── design-tokens.js           # Design token extraction
+│   │   ├── dom-tree-analyzer.js       # DOM hierarchy for structure analysis
+│   │   ├── dimension-extractor.js     # Component dimension measurement
+│   │   ├── section-cropper.js         # Section extraction for AI analysis
+│   │   ├── page-readiness.js          # Page stability detection
+│   │   ├── lazy-loader.js             # Lazy loading trigger & wait
+│   │   ├── cookie-handler.js          # Cookie banner dismissal
+│   │   ├── content-counter.js         # Content statistics
+│   │   ├── video-capture.js           # Scroll animation recording
+│   │   └── app-state-snapshot.js      # App state persistence
+│   ├── ai/                            # AI analysis modules
+│   │   ├── ux-audit.js                # UX audit runner
+│   │   └── prompts/                   # AI prompts
+│   ├── verification/                  # Verification scripts
+│   └── utils/                         # Shared utilities
+│       ├── browser.js                 # Browser abstraction facade
+│       ├── env.js                     # Environment resolution
+│       └── helpers.js                 # CLI utilities
+├── tests/                             # Unit tests
+│   ├── test-semantic-enhancer.js      # Semantic enhancer tests (59 tests)
+│   └── [other test files]
+└── package.json
+```
+
+## Key Modules
+
+### 1. semantic-enhancer.js (Phase 3)
+
+**Purpose**: Inject WordPress-compatible semantic IDs, classes, and ARIA roles into extracted HTML while preserving original styling.
+
+**Key Exports**:
+- `SEMANTIC_MAPPINGS` - Mapping definitions for header, nav, main, sidebar, footer, hero
+- `detectSectionType(element)` - Detect section type via semantic tags (priority 1), ARIA roles (priority 2), class patterns (priority 3)
+- `applySemanticAttributes(element, sectionType, options)` - Add ID/classes/roles to element
+- `handleMultipleNavs(navElements, usedIds)` - Handle multiple nav elements with aria-label
+- `enhanceSemanticHTML(html, domHierarchy)` - Browser-context enhancement (uses DOMParser)
+- `enhanceSemanticHTMLInPage(page, html)` - Playwright-context enhancement (recommended for Node.js)
+
+**Semantic Mappings**:
+```javascript
+header: { id: 'site-header', classes: ['site-header'], role: 'banner' }
+nav: { id: 'site-navigation', classes: ['main-navigation', 'nav-menu'], role: 'navigation' }
+main: { id: 'main-content', classes: ['site-main', 'content-area'], role: 'main' }
+sidebar: { id: 'primary-sidebar', classes: ['widget-area', 'sidebar'], role: 'complementary' }
+footer: { id: 'site-footer', classes: ['site-footer'], role: 'contentinfo' }
+hero: { id: 'hero-section', classes: ['hero'], role: null }
+```
+
+**Detection Priority**:
+1. Semantic HTML tags (header, nav, main, aside, footer)
+2. ARIA role attributes (banner, navigation, main, complementary, contentinfo)
+3. Class pattern matching (header, nav, main, sidebar, footer, hero)
+
+**Rules**:
+- Add ID only if none exists (avoid duplicates)
+- Append classes (never replace existing)
+- Set role only if not present
+- Handle multiple navs with proper aria-label (Primary Menu, Footer Menu, etc.)
+
+### 2. html-extractor.js (Modified)
+
+**New Function**: `extractAndEnhanceHtml(page, options)`
+
+Extracts clean HTML and optionally applies semantic enhancement via semantic-enhancer.js.
+
+**Options**:
+```javascript
+{
+  enhanceSemantic: true,           // Enable semantic enhancement (default: true)
+  frameworkPatterns: [...]         // Custom framework patterns to remove
+}
+```
+
+**Returns**:
+```javascript
+{
+  html: string,                    // Enhanced HTML
+  warnings: string[],              // Processing warnings
+  elementCount: number,            // DOM element count
+  semanticStats: {                 // Only if enhanceSemantic=true
+    sectionsEnhanced: number,
+    idsAdded: number,
+    classesAdded: number,
+    rolesAdded: number,
+    warnings: string[]
+  }
+}
+```
+
+**Existing Functions**:
+- `extractCleanHtml(page, frameworkPatterns)` - Remove scripts, event handlers, framework attributes
+
+### 3. screenshot.js (Modified)
+
+**New Flag**: `--no-semantic`
+
+Disable WordPress semantic HTML enhancement in extracted HTML. By default, semantic enhancement is enabled.
+
+**Usage**:
+```bash
+node src/core/screenshot.js --url https://example.com --output ./out --extract-html --no-semantic
+```
+
+### 4. multi-page-screenshot.js (Modified)
+
+Uses `extractAndEnhanceHtml()` instead of separate extraction steps.
+
+## Processing Pipeline
+
+### Multi-Viewport Screenshot Flow
+
+```
+Input URL
+├─ Desktop (1440x900)
+├─ Tablet (768x1024)
+└─ Mobile (375x812)
+      │
+      ├── Wait for page readiness (DOM stable, fonts loaded, styles stable)
+      ├── Dismiss cookie banners
+      ├── Trigger lazy loading
+      ├── Force lazy images visible
+      ├── Capture screenshots
+      │
+      ├── Optional: Extract HTML
+      │   ├─ Clean HTML (remove scripts, framework attrs)
+      │   └─ Semantic enhance (add WordPress IDs/classes/roles)
+      │
+      ├── Optional: Extract CSS
+      │   ├─ Collect all stylesheet rules
+      │   ├─ Extract @keyframes & transitions
+      │   └─ Filter unused selectors
+      │
+      ├── Optional: Capture hover states
+      │   ├─ Identify interactive elements
+      │   ├─ Screenshot before/during hover
+      │   └─ Generate :hover CSS rules
+      │
+      └── Output: Screenshots + metadata
+
+Output Files
+├── desktop.png, tablet.png, mobile.png
+├── source.html (cleaned + optionally semantically enhanced)
+├── source.css, source-raw.css
+├── animations.css, animation-tokens.json
+├── hover.css (if --capture-hover)
+├── structure.md (if GEMINI_API_KEY set)
+└── tokens.json
+```
+
+## Testing
+
+### Test Files
+
+- `tests/test-semantic-enhancer.js` - 59 unit tests covering:
+  - SEMANTIC_MAPPINGS exports
+  - Section type detection (header, nav, main, sidebar, footer, hero)
+  - Semantic attribute application
+  - Multiple nav handling with aria-labels
+  - HTML enhancement stats
+  - Page.evaluate integration
+
+**Run Tests**:
+```bash
+node tests/test-semantic-enhancer.js
+```
+
+## Data Flow
+
+### Semantic Enhancement Data Flow
+
+```
+extractAndEnhanceHtml()
+├─ extractCleanHtml(page)
+│  └─ page.evaluate()
+│     ├─ Clone document
+│     ├─ Remove scripts/noscript
+│     ├─ Remove malicious CSS links
+│     ├─ Remove event handlers
+│     ├─ Remove framework attributes
+│     ├─ Inline critical layout styles
+│     └─ Return cleaned HTML + warnings
+│
+└─ enhanceSemanticHTMLInPage(page, html)
+   └─ page.evaluate(enhancementLogic)
+      ├─ Parse HTML with DOMParser
+      ├─ Detect sections (semantic tags → ARIA roles → class patterns)
+      ├─ Apply IDs/classes/roles
+      ├─ Handle multiple navs with aria-labels
+      ├─ Detect hero sections
+      └─ Return enhanced HTML + stats
+```
+
+## Configuration & Environment
+
+### CLI Options (screenshot.js)
+
+| Option | Default | Phase | Description |
+|--------|---------|-------|-------------|
+| --url | required | - | Target URL |
+| --output | required | - | Output directory |
+| --viewports | all | - | Comma-separated viewport names |
+| --full-page | true | - | Capture full page height |
+| --max-size | 5 | - | Max file size (MB) before compression |
+| --headless | false | - | Run in headless mode |
+| --scroll-delay | 1500 | - | Pause time (ms) between scroll steps |
+| --extract-html | false | - | Extract cleaned HTML |
+| --extract-css | false | - | Extract CSS |
+| --filter-unused | true | - | Filter unused CSS selectors |
+| --capture-hover | false | 2 | Capture hover states |
+| --section-mode | false | - | Enable section-based capture |
+| --no-semantic | false | 3 | Disable semantic HTML enhancement |
+| --video | false | - | Record scroll animation |
+
+### Environment Variables
+
+```bash
+GEMINI_API_KEY=...      # For AI structure analysis
+```
+
+## Design Patterns
+
+### Error Handling
+
+All modules use try-catch with warning accumulation. Failed processing steps return partial results rather than throwing.
+
+### Idempotency
+
+Semantic enhancement is idempotent—running on already-enhanced HTML produces same result (IDs/classes/roles already present are skipped).
+
+### Performance
+
+- Combined landmark selector reduces querySelectorAll calls (8 → 1)
+- Processed element tracking prevents double-counting from overlapping selectors
+- Index-based element matching for reliability during DOM cloning
+
+### Validation
+
+- Input validation on HTML strings (non-empty, valid string type)
+- Browser context validation (DOMParser vs page.evaluate)
+- ID uniqueness tracking with usedIds Set
+- DOM size warnings (>50k elements)
+
+## Version History
+
+### Phase 1
+- Multi-viewport screenshots
+- HTML/CSS extraction
+- Asset extraction
+
+### Phase 2
+- Hover state capture
+- UX audit with Gemini
+- Design token extraction
+- DOM tree analysis
+
+### Phase 3
+- WordPress semantic HTML enhancement (CURRENT)
+- Semantic ID/class/role injection
+- ARIA landmark support
+- Multiple nav handling
+
+## Integration Points
+
+### With screenshot.js
+- New `--no-semantic` flag to disable enhancement
+- Automatic semantic enhancement when extracting HTML (unless disabled)
+
+### With html-extractor.js
+- New `extractAndEnhanceHtml()` function wraps extraction + enhancement
+- `enhanceSemantic` option controls semantic injection
+
+### With multi-page-screenshot.js
+- Uses `extractAndEnhanceHtml()` for HTML extraction
+
+## Dependencies
+
+- **playwright** - Browser automation
+- **sharp** - Image compression (optional)
+- **google-genai** - AI analysis (optional, for Phase 2 features)
+
+## Limitations & Considerations
+
+1. **Browser Context Required**: `enhanceSemanticHTML()` requires DOMParser (browser). Use `enhanceSemanticHTMLInPage()` for Playwright.
+2. **Non-Invasive**: Semantic enhancement never removes existing attributes, only adds/appends.
+3. **False Positive Prevention**: Class pattern detection limited to container elements (div, section, article) to avoid false positives.
+4. **Multiple Landing Pages**: Each nav gets unique aria-label (Primary Menu, Footer Menu, Navigation 2, etc.)
+5. **Hero Section Detection**: Only top-level hero elements (not within header/footer) are detected.