dembrandt 0.3.0 → 0.5.0

package/README.md

@@ -1,420 +1,123 @@
-# 🎨 Dembrandt
+# Dembrandt.
 
 [![npm version](https://img.shields.io/npm/v/dembrandt.svg)](https://www.npmjs.com/package/dembrandt)
 [![npm downloads](https://img.shields.io/npm/dm/dembrandt.svg)](https://www.npmjs.com/package/dembrandt)
-[![license](https://img.shields.io/npm/l/dembrandt.svg)](https://github.com/thevangelist/dembrandt/blob/main/LICENSE)
+[![license](https://img.shields.io/npm/l/dembrandt.svg)](https://github.com/dembrandt/dembrandt/blob/main/LICENSE)
 
-A CLI tool for extracting design tokens and brand assets from any website. Powered by Playwright with advanced bot detection avoidance.
+Extract any website’s design system into design tokens in a few seconds: logo, colors, typography, borders, and more. One command.
 
 ![Dembrandt Demo](showcase.png)
 
-## Quick Start
+## Install
 
 ```bash
-npx dembrandt stripe.com
+npx dembrandt bmw.de
 ```
 
-No installation required! Extract design tokens from any website in seconds. Or install globally with `npm install -g dembrandt`.
+Or install globally: `npm install -g dembrandt` then run `dembrandt bmw.de`
 
-## What It Does
+Requires Node.js 18+
 
-Dembrandt analyzes live websites and extracts their complete design system:
+## What to expect from extraction?
 
-- **Logo** — Logo detection (img/svg) with dimensions and source URL
-- **Favicons** — All favicon variants with sizes and types
-- **Colors** — Semantic colors, color palette with confidence scoring, CSS variables (both hex and RGB formats)
-- **Typography** — Font families, sizes, weights, line heights, font sources (Google Fonts, Adobe Fonts, custom)
-- **Spacing** — Margin and padding scales with grid system detection (4px/8px/custom)
-- **Border Radius** — Corner radius patterns with usage frequency
-- **Borders** — Border widths, styles (solid, dashed, dotted), and colors with confidence scoring
-- **Shadows** — Box shadow values for elevation systems
-- **Buttons** — Component styles with variants and states
-- **Inputs** — Form field styles (input, textarea, select)
-- **Links** — Link styles with hover states and decorations
-- **Breakpoints** — Responsive design breakpoints from media queries
-- **Icons** — Icon system detection (Font Awesome, Material Icons, SVG)
-- **Frameworks** — CSS framework detection (Tailwind, Bootstrap, Material-UI, Chakra)
-
-Perfect for competitive analysis, brand audits, or rebuilding a brand when you don't have design guidelines.
-
-## Why It Matters
-
-**Designers** — Analyze competitor systems, document production tokens, audit brand consistency
-
-**Developers** — Migrate design tokens, reverse engineer components, validate implementations
-
-**Product Managers** — Track competitor evolution, quantify design debt, evaluate vendors
-
-**Marketing** — Audit competitor brands, plan rebrands, monitor brand compliance
-
-**Engineering Leaders** — Measure technical debt, plan migrations, assess acquisition targets
-
-## How It Works
-
-Uses Playwright to render the page, extracts computed styles from the DOM, analyzes color usage and confidence, groups similar typography, detects spacing patterns, and returns actionable design tokens.
-
-### Extraction Process
-
-1. **Browser Launch** - Launches Chromium with stealth configuration
-2. **Anti-Detection** - Injects scripts to bypass bot detection
-3. **Navigation** - Navigates to target URL with retry logic
-4. **Hydration** - Waits for SPAs to fully load (8s initial + 4s stabilization)
-5. **Content Validation** - Verifies page content is substantial (>500 chars)
-6. **Parallel Extraction** - Runs all extractors concurrently for speed
-7. **Analysis** - Analyzes computed styles, DOM structure, and CSS variables
-8. **Scoring** - Assigns confidence scores based on context and usage
-
-### Color Confidence
-
-- **High** — Logo, brand elements, primary buttons
-- **Medium** — Interactive elements, icons, navigation
-- **Low** — Generic UI components (filtered from display)
-
-Only shows high and medium confidence colors in terminal. Full palette in JSON.
-
-### Typography Detection
-
-Samples all heading levels (h1-h6), body text, buttons, links. Groups by font family, size, and weight. Detects Google Fonts, Adobe Fonts, custom @font-face.
-
-### Framework Detection
-
-Recognizes Tailwind CSS, Bootstrap, Material-UI, and others by class patterns and CDN links.
-
-## Installation
-
-### Using npx (Recommended)
-
-No installation needed! Run directly with `npx`:
-
-```bash
-npx dembrandt stripe.com
-```
-
-The first run will automatically install Chromium (~170MB).
-
-### Global Installation
-
-Install globally for repeated use:
-
-```bash
-npm install -g dembrandt
-dembrandt stripe.com
-```
-
-### Prerequisites
-
-- Node.js 18 or higher
-
-### Development Setup
-
-For contributors who want to work on dembrandt:
-
-```bash
-git clone https://github.com/thevangelist/dembrandt.git
-cd dembrandt
-npm install
-npm link
-```
+- Colors (semantic, palette, CSS variables)
+- Typography (fonts, sizes, weights, sources)
+- Spacing (margin/padding scales)
+- Borders (radius, widths, styles, colors)
+- Shadows
+- Components (buttons, badges, inputs, links)
+- Breakpoints
+- Icons & frameworks
 
 ## Usage
 
-### Basic Usage
-
 ```bash
-# Using npx (no installation)
-npx dembrandt <url>
-
-# Or if installed globally
-dembrandt <url>
-
-# Examples
-dembrandt stripe.com
-dembrandt https://github.com
-dembrandt tailwindcss.com
+dembrandt <url>                    # Basic extraction (terminal display only)
+dembrandt bmw.de --json-only       # Output raw JSON to terminal (no formatted display, no file save)
+dembrandt bmw.de --save-output     # Save JSON to output/bmw.de/YYYY-MM-DDTHH-MM-SS.json
+dembrandt bmw.de --dtcg            # Export in W3C Design Tokens (DTCG) format (auto-saves as .tokens.json)
+dembrandt bmw.de --dark-mode       # Extract colors from dark mode variant
+dembrandt bmw.de --mobile          # Use mobile viewport (390x844, iPhone 12/13/14/15) for responsive analysis
+dembrandt bmw.de --slow            # 3x longer timeouts (24s hydration) for JavaScript-heavy sites
+dembrandt bmw.de --no-sandbox      # Disable Chromium sandbox (required for Docker/CI)
 ```
 
-### Options
+Default: formatted terminal display only. Use `--save-output` to persist results as JSON files. Browser automatically retries in visible mode if headless extraction fails.
 
-**`--json-only`** - Output raw JSON to stdout instead of formatted terminal display
+### W3C Design Tokens (DTCG) Format
 
-```bash
-dembrandt stripe.com --json-only > tokens.json
-```
-
-Note: JSON is automatically saved to `output/domain.com/` regardless of this flag.
-
-**`-d, --debug`** - Run with visible browser and detailed logs
-
-```bash
-dembrandt stripe.com --debug
-```
-
-Useful for troubleshooting bot detection, timeouts, or extraction issues.
-
-**`--verbose-colors`** - Show medium and low confidence colors in terminal output
+Use `--dtcg` to export in the standardized [W3C Design Tokens Community Group](https://www.designtokens.org/) format:
 
 ```bash
-dembrandt stripe.com --verbose-colors
+dembrandt stripe.com --dtcg
+# Saves to: output/stripe.com/TIMESTAMP.tokens.json
 ```
 
-By default, only high-confidence colors are shown. Use this flag to see all detected colors.
-
-**`--dark-mode`** - Extract colors from dark mode
+Transform DTCG tokens to platform-specific formats using [Style Dictionary](https://styledictionary.com):
 
 ```bash
-dembrandt stripe.com --dark-mode
+# Generate CSS variables, SCSS, JavaScript, TypeScript
+npm run build:tokens
 ```
 
-Enables dark mode preference detection for sites that support it.
-
-**`--mobile`** - Extract from mobile viewport
-
-```bash
-dembrandt stripe.com --mobile
-```
-
-Simulates a mobile device viewport for responsive design token extraction.
-
-**`--slow`** - Use 3x longer timeouts for slow-loading sites
-
-```bash
-dembrandt linkedin.com --slow
-```
-
-Helpful for sites with heavy JavaScript, complex SPAs, or aggressive bot detection that need extra time to fully load.
-
-## Output
-
-### Automatic JSON Saves
-
-Every extraction is automatically saved to `output/domain.com/YYYY-MM-DDTHH-MM-SS.json` with:
-
-- Complete design token data
-- Timestamped for version tracking
-- Organized by domain
-
-Example: `output/stripe.com/2025-11-22T14-30-45.json`
-
-### Terminal Output
-
-Clean, formatted tables showing:
-
-- Color palette with confidence ratings (with visual swatches)
-- CSS variables with color previews
-- Typography hierarchy with context
-- Spacing scale (4px/8px grid detection)
-- Shadow system
-- Button variants
-- Component style breakdowns
-- Framework and icon system detection
-
-### JSON Output Format
-
-Complete extraction data for programmatic use:
-
-```json
-{
-  "url": "https://example.com",
-  "extractedAt": "2025-11-22T...",
-  "logo": { "source": "img", "url": "...", "width": 120, "height": 40 },
-  "colors": {
-    "semantic": { "primary": "#3b82f6", ... },
-    "palette": [{ "color": "#3b82f6", "confidence": "high", "count": 45, "sources": [...] }],
-    "cssVariables": { "--color-primary": "#3b82f6", ... }
-  },
-  "typography": {
-    "styles": [{ "fontFamily": "Inter", "fontSize": "16px", "fontWeight": "400", ... }],
-    "sources": { "googleFonts": [...], "adobeFonts": false, "customFonts": [...] }
-  },
-  "spacing": { "scaleType": "8px", "commonValues": [{ "px": "16px", "rem": "1rem", "count": 42 }, ...] },
-  "borderRadius": { "values": [{ "value": "8px", "count": 15, "confidence": "high" }, ...] },
-  "shadows": [{ "shadow": "0 2px 4px rgba(0,0,0,0.1)", "count": 8, "confidence": "high" }, ...],
-  "components": {
-    "buttons": [{ "backgroundColor": "...", "color": "...", "padding": "...", ... }],
-    "inputs": [{ "type": "input", "border": "...", "borderRadius": "...", ... }]
-  },
-  "breakpoints": [{ "px": "768px" }, ...],
-  "iconSystem": [{ "name": "Font Awesome", "type": "icon-font" }, ...],
-  "frameworks": [{ "name": "Tailwind CSS", "confidence": "high", "evidence": "class patterns" }]
-}
-```
-
-## Examples
-
-### Extract Design Tokens
-
-```bash
-# Analyze a single site (auto-saves JSON to output/stripe.com/)
-dembrandt stripe.com
-
-# View saved JSON files
-ls output/stripe.com/
-
-# Output to stdout for piping
-dembrandt stripe.com --json-only | jq '.colors.semantic'
-
-# Debug mode for difficult sites
-dembrandt example.com --debug
-```
-
-### Compare Competitors
-
-```bash
-# Extract tokens from multiple competitors (auto-saved to output/)
-for site in stripe.com square.com paypal.com; do
-  dembrandt $site
-done
-
-# Compare color palettes from most recent extractions
-jq '.colors.palette[] | select(.confidence=="high")' output/stripe.com/2025-11-22T*.json output/square.com/2025-11-22T*.json
-
-# Compare semantic colors across competitors
-jq '.colors.semantic' output/*/2025-11-22T*.json
-```
-
-### Integration with Design Tools
-
-```bash
-# Extract and convert to custom config format
-dembrandt mysite.com --json-only | jq '{
-  colors: .colors.semantic,
-  fontFamily: .typography.sources,
-  spacing: .spacing.commonValues
-}' > design-tokens.json
-
-# Or use the built-in Tailwind CSS exporter (see lib/exporters.js)
-# Converts extracted tokens to Tailwind config format
-```
+Outputs CSS custom properties, SCSS variables, ES6 modules, and TypeScript definitions. See [STYLE-DICTIONARY.md](STYLE-DICTIONARY.md) for details.
 
 ## Use Cases
 
-### Brand Audits
-
-Extract and document your company's current design system from production websites.
-
-### Competitive Analysis
-
-Compare design systems across competitors to identify trends and opportunities.
-
-### Design System Migration
+- Brand audits & competitive analysis
+- Design system documentation
+- Reverse engineering brands
+- Multi-site brand consolidation
 
-Document legacy design tokens before migrating to a new system.
-
-### Reverse Engineering
-
-Rebuild a brand when original design guidelines are unavailable.
-
-### Quality Assurance
-
-Verify design consistency across different pages and environments.
-
-## Advanced Features
-
-### Bot Detection Avoidance
-
-- Stealth mode with anti-detection scripts
-- Automatic fallback to visible browser on detection
-- Human-like interaction simulation (mouse movement, scrolling)
-- Custom user agent and browser fingerprinting
-
-### Smart Retry Logic
-
-- Automatic retry on navigation failures (up to 2 attempts)
-- SPA hydration detection and waiting
-- Content validation to ensure page is fully loaded
-- Detailed progress logging at each step
-
-### Comprehensive Logging
-
-- Real-time spinner with step-by-step progress
-- Detailed extraction metrics (colors found, styles detected, etc.)
-- Error context with URL, stage, and attempt information
-- Debug mode with full stack traces
-
-## Troubleshooting
-
-### Bot Detection Issues
-
-If you encounter timeouts or network errors:
-
-```bash
-dembrandt example.com --debug
-```
-
-This will automatically retry with a visible browser.
-
-### Page Not Loading
-
-Some sites require longer load times. The tool waits 8 seconds for SPA hydration, but you can modify this in the source.
+## How It Works
 
-### Empty Content
+Uses Playwright to render the page, extracts computed styles from the DOM, analyzes color usage and confidence, groups similar typography, detects spacing patterns, and returns actionable design tokens.
 
-If content length is < 500 chars, the tool will automatically retry (up to 2 attempts).
+### Extraction Process
 
-### Debug Mode
+1. Browser Launch - Launches Chromium with stealth configuration
+2. Anti-Detection - Injects scripts to bypass bot detection
+3. Navigation - Navigates to target URL with retry logic
+4. Hydration - Waits for SPAs to fully load (8s initial + 4s stabilization)
+5. Content Validation - Verifies page content is substantial (>500 chars)
+6. Parallel Extraction - Runs all extractors concurrently for speed
+7. Analysis - Analyzes computed styles, DOM structure, and CSS variables
+8. Scoring - Assigns confidence scores based on context and usage
 
-Use `--debug` to see:
+### Color Confidence
 
-- Browser launch confirmation
-- Step-by-step progress logs
-- Full error stack traces
-- Extraction metrics
+- High — Logo, brand elements, primary buttons
+- Medium — Interactive elements, icons, navigation
+- Low — Generic UI components (filtered from display)
+- Only shows high and medium confidence colors in terminal. Full palette in JSON.
 
 ## Limitations
 
-- Dark mode requires `--dark-mode` flag (not automatically detected)
+- Dark mode requires --dark-mode flag (not automatically detected)
 - Hover/focus states extracted from CSS (not fully interactive)
 - Canvas/WebGL-rendered sites cannot be analyzed (e.g., Tesla, Apple Vision Pro demos)
 - JavaScript-heavy sites require hydration time (8s initial + 4s stabilization)
 - Some dynamically-loaded content may be missed
-- Default viewport is 1920x1080 (use `--mobile` for responsive analysis)
-
-## Architecture
-
-```
-dembrandt/
-├── index.js                      # CLI entry point, command handling
-├── lib/
-│   ├── extractors.js             # Core extraction logic with stealth mode
-│   ├── display.js                # Terminal output formatting
-│   └── exporters.js              # Export to Tailwind CSS config (NEW)
-├── output/                       # Auto-saved JSON extractions (gitignored)
-│   ├── stripe.com/
-│   │   ├── 2025-11-22T14-30-45.json
-│   │   └── 2025-11-22T15-12-33.json
-│   └── github.com/
-│       └── 2025-11-22T14-35-12.json
-├── package.json
-└── README.md
-```
+- Default viewport is 1920x1080 (use --mobile for 390x844 iPhone viewport)
 
 ## Ethics & Legality
 
 Dembrandt extracts publicly available design information (colors, fonts, spacing) from website DOMs for analysis purposes. This falls under fair use in most jurisdictions (USA's DMCA § 1201(f), EU Software Directive 2009/24/EC) when used for competitive analysis, documentation, or learning.
 
-**Legal:** Analyzing public HTML/CSS is generally legal. Does not bypass protections or violate copyright. Check site ToS before mass extraction.
+Legal: Analyzing public HTML/CSS is generally legal. Does not bypass protections or violate copyright. Check site ToS before mass extraction.
 
-**Ethical:** Use for inspiration and analysis, not direct copying. Respect servers (no mass crawling), give credit to sources, be transparent about data origin.
+Ethical: Use for inspiration and analysis, not direct copying. Respect servers (no mass crawling), give credit to sources, be transparent about data origin.
 
 ## Contributing
 
-Issues and pull requests welcome. Please include:
+Bugs you found? Weird websites that make it cry? Pull requests (even one-liners make me happy)?
 
-- Clear description of the issue/feature
-- Example URLs that demonstrate the problem
-- Expected vs actual behavior
+Spam me in [Issues](https://github.com/dembrandt/dembrandt/issues) or PRs. I reply to everything.
 
-## License
+Let's keep the light alive together.
 
-MIT
+@thevangelist
 
-## Roadmap
+---
 
-- [x] Dark mode extraction (via `--dark-mode` flag)
-- [x] Mobile viewport support (via `--mobile` flag)
-- [x] Clickable terminal links for modern terminals
-- [?] Figma integration branch: `figma-integration`
-- [ ] Animation/transition detection
-- [ ] Interactive state capture (hover, focus, active)
-- [ ] Multi-page analysis
-- [ ] Configuration file support
+MIT — do whatever you want with it.

package/index.js

@@ -10,55 +10,49 @@
 import { program } from "commander";
 import chalk from "chalk";
 import ora from "ora";
-import { chromium } from "playwright";
+import { chromium } from "playwright-core";
 import { extractBranding } from "./lib/extractors.js";
 import { displayResults } from "./lib/display.js";
+import { toW3CFormat } from "./lib/w3c-exporter.js";
 import { writeFileSync, mkdirSync } from "fs";
 import { join } from "path";
 
 program
   .name("dembrandt")
   .description("Extract design tokens from any website")
-  .version("1.0.0")
+  .version("0.5.0")
   .argument("<url>")
   .option("--json-only", "Output raw JSON")
-  .option("-d, --debug", "Force visible browser")
-  .option("--verbose-colors", "Show medium and low confidence colors")
+  .option("--save-output", "Save JSON file to output folder")
+  .option("--dtcg", "Export in W3C Design Tokens (DTCG) format")
   .option("--dark-mode", "Extract colors from dark mode")
   .option("--mobile", "Extract from mobile viewport")
   .option("--slow", "3x longer timeouts for slow-loading sites")
+  .option("--no-sandbox", "Disable browser sandbox (needed for Docker/CI)")
   .action(async (input, opts) => {
     let url = input;
-    if (!url.match(/^https?:\/\//)) url = "https://" + url;
+    if (!url.startsWith("http://") && !url.startsWith("https://")) {
+      url = "https://" + url;
+    }
 
     const spinner = ora("Starting extraction...").start();
     let browser = null;
 
     try {
-      let useHeaded = opts.debug;
+      let useHeaded = false;
       let result;
 
       while (true) {
-        spinner.text = `Launching browser (${
-          useHeaded ? "visible" : "headless"
-        } mode)`;
+        spinner.text = `Launching browser (${useHeaded ? "visible" : "headless"
+          } mode)`;
+        const launchArgs = ["--disable-blink-features=AutomationControlled"];
+        if (opts.noSandbox) {
+          launchArgs.push("--no-sandbox", "--disable-setuid-sandbox");
+        }
         browser = await chromium.launch({
           headless: !useHeaded,
-          args: [
-            "--no-sandbox",
-            "--disable-setuid-sandbox",
-            "--disable-blink-features=AutomationControlled",
-          ],
+          args: launchArgs,
         });
-        if (opts.debug) {
-          console.log(
-            chalk.dim(
-              `  ✓ Browser launched in ${
-                useHeaded ? "visible" : "headless"
-              } mode`
-            )
-          );
-        }
 
         try {
           result = await extractBranding(url, spinner, browser, {
@@ -93,8 +87,11 @@ program
 
       console.log();
 
-      // Save JSON output automatically (unless --json-only)
-      if (!opts.jsonOnly) {
+      // Convert to W3C format if requested
+      const outputData = opts.dtcg ? toW3CFormat(result) : result;
+
+      // Save JSON output if --save-output or --dtcg is specified
+      if ((opts.saveOutput || opts.dtcg) && !opts.jsonOnly) {
         try {
           const domain = new URL(url).hostname.replace("www.", "");
           const timestamp = new Date()
@@ -105,9 +102,10 @@ program
           const outputDir = join(process.cwd(), "output", domain);
           mkdirSync(outputDir, { recursive: true });
 
-          const filename = `${timestamp}.json`;
+          const suffix = opts.dtcg ? '.tokens' : '';
+          const filename = `${timestamp}${suffix}.json`;
           const filepath = join(outputDir, filename);
-          writeFileSync(filepath, JSON.stringify(result, null, 2));
+          writeFileSync(filepath, JSON.stringify(outputData, null, 2));
 
           console.log(
             chalk.dim(
@@ -125,29 +123,16 @@ program
 
       // Output to terminal
       if (opts.jsonOnly) {
-        console.log(JSON.stringify(result, null, 2));
+        console.log(JSON.stringify(outputData, null, 2));
       } else {
         console.log();
-        displayResults(result, { verboseColors: opts.verboseColors });
+        displayResults(result);
       }
     } catch (err) {
       spinner.fail("Failed");
       console.error(chalk.red("\n✗ Extraction failed"));
       console.error(chalk.red(`  Error: ${err.message}`));
       console.error(chalk.dim(`  URL: ${url}`));
-
-      if (opts.debug && err.stack) {
-        console.error(chalk.dim("\nStack trace:"));
-        console.error(chalk.dim(err.stack));
-      }
-
-      if (!opts.debug) {
-        console.log(
-          chalk.hex('#FFB86C')(
-            "\nTip: Try with --debug flag for tough sites and detailed error logs"
-          )
-        );
-      }
       process.exit(1);
     } finally {
       if (browser) await browser.close();