smippo 0.0.9 → 0.1.0

package/README.md

@@ -27,45 +27,6 @@
 
 📚 **[View complete documentation →](https://smippo.com)**
 
-## Table of Contents
-
-- [Table of Contents](#table-of-contents)
-- [Features](#features)
-- [Quick Start](#quick-start)
-- [Installation](#installation)
-  - [Requirements](#requirements)
-  - [npm (Global)](#npm-global)
-  - [Homebrew (Coming soon)](#homebrew-coming-soon)
-- [Usage](#usage)
-  - [Basic Usage](#basic-usage)
-  - [Interactive Mode](#interactive-mode)
-  - [Filtering](#filtering)
-  - [Scope Control](#scope-control)
-  - [Browser Options](#browser-options)
-  - [Screenshots](#screenshots)
-  - [Authentication](#authentication)
-  - [Output Options](#output-options)
-  - [Performance \& Parallelism: The Vacuum Architecture](#performance--parallelism-the-vacuum-architecture)
-  - [Continue/Update](#continueupdate)
-  - [Serve](#serve)
-  - [Static Mode](#static-mode)
-- [Structured Output](#structured-output)
-- [Programmatic API](#programmatic-api)
-- [Contributing](#contributing)
-- [License](#license)
-- [Acknowledgments](#acknowledgments)
-
-## Features
-
-- **🚀 Vacuum Architecture** — Parallel workers consume sites rapidly, just like hippos vacuum up everything in their path
-- **📸 Structured Mirroring** — Every page, every resource, every network request captured in organized, structured output
-- **🔍 Complete Fidelity** — Gets the page exactly as you see it, including CSS-in-JS, dynamic content, and lazy-loaded images
-- **🎯 Smart Consumption** — Respects robots.txt, filters by URL patterns, MIME types, and file sizes
-- **📦 Structured Output** — Organized mirror structure preserves original paths for seamless offline browsing
-- **🎨 Beautiful CLI** — Interactive guided mode, progress bars, and elegant terminal output
-- **🌐 Built-in Server** — Serve captured sites locally with directory browsing
-- **📊 HAR Files** — Generates HTTP Archive files for debugging and replay
-
 ## Quick Start
 
 Install globally:
@@ -92,336 +53,48 @@ Or use without installing:
 npx smippo https://example.com
 ```
 
-> 📖 **For complete documentation, guides, and API reference, visit [smippo.com](https://smippo.com)**
-
-## Installation
-
-### Requirements
-
-- Node.js 18 or later
-- Chromium (automatically downloaded on first install)
-
-### npm (Global)
-
-```bash
-npm install -g smippo
-```
-
-### Homebrew (Coming soon)
-
-```bash
-brew install smippo
-```
-
-## Usage
-
-### Basic Usage
-
-```bash
-# Capture a single page with all assets
-smippo https://example.com
-
-# Mirror a site with depth control
-smippo https://example.com --depth 3
-
-# Save to custom directory
-smippo https://example.com --output ./my-mirror
-```
-
-### Interactive Mode
-
-Just run `smippo` with no arguments to start the guided wizard:
-
-```bash
-smippo
-```
-
-This will walk you through:
-
-- URL to capture
-- Crawl depth
-- Scope settings
-- Asset options
-- Advanced configuration
-
-Perfect for beginners or when you want to explore options!
-
-### Filtering
-
-```bash
-# Include only specific patterns
-smippo https://example.com --include "*.html" --include "*.css"
-
-# Exclude patterns
-smippo https://example.com --exclude "*tracking*" --exclude "*ads*"
-
-# Filter by MIME type
-smippo https://example.com --mime-include "image/*" --mime-exclude "video/*"
-
-# Filter by file size
-smippo https://example.com --max-size 5MB --min-size 1KB
-```
-
-### Scope Control
-
-```bash
-# Stay on same subdomain (default)
-smippo https://www.example.com --scope subdomain
-
-# Allow all subdomains
-smippo https://www.example.com --scope domain
-
-# Go everywhere (use with caution!)
-smippo https://example.com --scope all --depth 2
-```
-
-### Browser Options
+## Commands
 
-```bash
-# Wait for specific condition
-smippo https://example.com --wait networkidle
-smippo https://example.com --wait domcontentloaded
-
-# Add extra wait time for slow sites
-smippo https://example.com --wait-time 5000
-
-# Custom user agent
-smippo https://example.com --user-agent "Mozilla/5.0..."
+Smippo provides several commands for different use cases:
 
-# Custom viewport
-smippo https://example.com --viewport 1280x720
-
-# Emulate device
-smippo https://example.com --device "iPhone 13"
-```
-
-### Screenshots
-
-Take quick screenshots without mirroring the full site:
-
-```bash
-# Basic screenshot
-smippo capture https://example.com
+- **`smippo <url>`** — Capture and mirror websites with full fidelity
+- **`smippo capture <url>`** — Take screenshots of web pages
+- **`smippo serve <directory>`** — Serve captured sites locally
+- **`smippo continue`** — Resume an interrupted capture
+- **`smippo update`** — Update an existing mirror
 
-# Full-page screenshot (captures entire scrollable page)
-smippo capture https://example.com --full-page
+Run `smippo` with no arguments to start the interactive guided mode.
 
-# Save to specific file
-smippo capture https://example.com -O ./screenshots/example.png
-
-# Mobile device screenshot
-smippo capture https://example.com --device "iPhone 13" -O mobile.png
-
-# Screenshot with dark mode
-smippo capture https://example.com --dark-mode
-
-# Capture specific element
-smippo capture https://example.com --selector ".hero-section"
-
-# JPEG format with quality
-smippo capture https://example.com --format jpeg --quality 90
-```
-
-### Authentication
-
-```bash
-# Basic auth
-smippo https://user:pass@example.com
-
-# Cookie-based auth
-smippo https://example.com --cookies cookies.json
-
-# Interactive login (opens browser window)
-smippo https://example.com --capture-auth
-```
-
-### Output Options
-
-```bash
-# Generate screenshots
-smippo https://example.com --screenshot
-
-# Generate PDFs
-smippo https://example.com --pdf
-
-# Skip HAR file
-smippo https://example.com --no-har
-
-# Output structure
-smippo https://example.com --structure original  # URL paths (default)
-smippo https://example.com --structure flat      # All in one directory
-smippo https://example.com --structure domain    # Organized by domain
-```
-
-### Performance & Parallelism: The Vacuum Architecture
-
-Smippo's parallel worker architecture mirrors how hippos consume everything in their path—rapidly and efficiently. Multiple workers operate simultaneously, each vacuuming up pages, resources, and network requests in parallel.
-
-```bash
-# Default: 8 parallel workers (8 hippos vacuuming simultaneously)
-smippo https://example.com
-
-# Limit to 4 workers (for rate-limited sites)
-smippo https://example.com --workers 4
-
-# Single worker (sequential, safest)
-smippo https://example.com --workers 1
-
-# Maximum speed (use with caution)
-smippo https://example.com --workers 16
-
-# Limit total pages
-smippo https://example.com --max-pages 100
-
-# Limit total time
-smippo https://example.com --max-time 300  # 5 minutes
-
-# Rate limiting (delay between requests per worker)
-smippo https://example.com --rate-limit 1000  # 1 second between requests
-```
-
-**The Vacuum Architecture:**
-
-Each worker operates like an independent hippo, vacuuming up:
-
-- Fully rendered pages (after JavaScript execution)
-- All network resources (images, fonts, stylesheets, API responses)
-- Network metadata (captured in HAR files)
-- Link structures (for recursive crawling)
-
-All captured content is then **structured** into organized mirrors that preserve original paths and relationships.
-
-**Tips for optimal performance:**
-
-- Use `--workers 1` for sites with strict rate limiting
-- Use `--workers 4-8` for most sites (default: 8)
-- Use `--workers 16` only for fast servers you control
-- Combine `--workers` with `--rate-limit` for polite crawling
-
-### Continue/Update
-
-```bash
-# Continue an interrupted capture
-smippo continue
-
-# Update an existing mirror
-smippo update
-```
-
-### Serve
-
-Serve captured sites locally with a built-in web server:
-
-```bash
-# Serve with auto port detection
-smippo serve ./site
-
-# Specify port
-smippo serve ./site --port 3000
-
-# Open browser automatically
-smippo serve ./site --open
-
-# Show all requests
-smippo serve ./site --verbose
-```
-
-The server provides:
-
-- **Auto port detection** — Finds next available port if default is busy
-- **Proper MIME types** — Correct content-type headers for all file types
-- **CORS support** — Enabled by default for local development
-- **Nice terminal UI** — Shows clickable URL and request logs
-
-### Static Mode
-
-For any site, use `--static` to strip scripts for true offline viewing:
-
-```bash
-# Capture as static HTML (removes JS, keeps rendered content)
-smippo https://example.com --static --external-assets
-
-# Then serve
-smippo serve ./site --open
-```
-
-## Structured Output
-
-Smippo creates **structured mirrors** that preserve the original URL structure and relationships. Every page, every resource, every network request is organized and stored in a logical hierarchy:
-
-```
-site/
-├── example.com/
-│   ├── index.html
-│   ├── about/
-│   │   └── index.html
-│   └── assets/
-│       ├── style.css
-│       └── logo.png
-├── .smippo/
-│   ├── cache.json          # Metadata cache
-│   ├── network.har         # HAR file
-│   ├── manifest.json       # Capture manifest
-│   └── log.txt             # Capture log
-└── index.html              # Entry point
-```
-
-## Programmatic API
-
-```javascript
-import {capture, Crawler, createServer} from 'smippo';
-
-// Simple capture
-const result = await capture('https://example.com', {
-  output: './mirror',
-  depth: 2,
-});
-
-console.log(`Captured ${result.stats.pagesCapt} pages`);
-
-// Advanced usage with events
-const crawler = new Crawler({
-  url: 'https://example.com',
-  output: './mirror',
-  depth: 3,
-  scope: 'domain',
-});
-
-crawler.on('page:complete', ({url, size}) => {
-  console.log(`Captured: ${url} (${size} bytes)`);
-});
+## Features
 
-crawler.on('error', ({url, error}) => {
-  console.error(`Failed: ${url} - ${error.message}`);
-});
+- **🚀 Vacuum Architecture** — Parallel workers consume sites rapidly
+- **📸 Complete Fidelity** — Captures pages exactly as rendered, including CSS-in-JS, dynamic content, and lazy-loaded images
+- **🎯 Smart Filtering** — Filter by URL patterns, MIME types, and file sizes. Respects robots.txt
+- **🌐 Built-in Server** — Serve captured sites locally with directory browsing
+- **📊 HAR Files** — Generates HTTP Archive files for debugging and replay
+- **💻 Programmatic API** — Use Smippo in your Node.js applications
 
-await crawler.start();
+## Documentation
 
-// Start a server programmatically
-const server = await createServer({
-  directory: './mirror',
-  port: 8080,
-  open: true, // Opens browser automatically
-});
+For complete documentation, guides, and API reference, visit **[smippo.com](https://smippo.com)**:
 
-console.log(`Server running at ${server.url}`);
+- **[Installation Guide](https://smippo.com/getting-started/installation)** — Detailed installation instructions
+- **[Commands Reference](https://smippo.com/commands)** — All available commands and options
+- **[Configuration](https://smippo.com/configuration)** — Filtering, scope control, performance tuning
+- **[Guides](https://smippo.com/guides)** — Output structure, link rewriting, troubleshooting
+- **[Programmatic API](https://smippo.com/api/programmatic)** — Use Smippo in your Node.js code
+- **[Examples](https://smippo.com/getting-started/examples)** — Real-world use cases
 
-// Later: stop the server
-await server.close();
-```
+## Requirements
 
-> 📖 **For complete API documentation, see the [Programmatic API guide](https://smippo.com/api-reference/programmatic-api) on smippo.com**
+- Node.js 18 or later
+- Chromium (automatically downloaded on first install)
 
 ## Contributing
 
 Contributions are welcome! Whether it's bug reports, feature requests, or pull requests — all contributions help make Smippo better.
 
-Please read our [Contributing Guide](CONTRIBUTING.md) for details on:
-
-- Development setup
-- Code style guidelines
-- Pull request process
-- Testing requirements
+Please read our [Contributing Guide](CONTRIBUTING.md) for details on development setup, code style guidelines, and the pull request process.
 
 Quick start:
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "smippo",
-  "version": "0.0.9",
+  "version": "0.1.0",
   "description": "S.M.I.P.P.O. — Structured Mirroring of Internet Pages and Public Objects. Modern website copier that captures sites exactly as they appear in your browser.",
   "main": "src/index.js",
   "bin": {

package/src/cli.js

@@ -10,6 +10,12 @@ import {
   runInteractiveCapture,
   shouldRunInteractive,
 } from './interactive.js';
+import {
+  getSiteDir,
+  getDomainFromUrl,
+  addSiteToGlobalManifest,
+  ensureSmippoHome,
+} from './utils/home.js';
 
 const program = new Command();
 
@@ -52,7 +58,10 @@ export function run() {
   // Main capture command
   program
     .argument('[url]', 'URL to capture')
-    .option('-o, --output <dir>', 'Output directory', './site')
+    .option(
+      '-o, --output <dir>',
+      'Output directory (default: ~/.smippo/sites/[domain])',
+    )
     .option('-d, --depth <n>', 'Recursion depth (0 = single page)', '0')
     .option('--no-crawl', 'Disable link following (same as -d 0)')
     .option('--dry-run', 'Show what would be captured without downloading')
@@ -179,7 +188,7 @@ export function run() {
   // Serve command
   program
     .command('serve [directory]')
-    .description('Serve a captured site locally')
+    .description('Serve a captured site locally (default: ~/.smippo/sites/)')
     .option(
       '-p, --port <port>',
       'Port to serve on (auto-finds available)',
@@ -192,8 +201,13 @@ export function run() {
     .option('-q, --quiet', 'Minimal output')
     .action(async (directory, options) => {
       const {serve} = await import('./server.js');
+      const {getSitesDir} = await import('./utils/home.js');
+
+      // If no directory specified, use global smippo sites directory
+      const serveDir = directory || getSitesDir();
+
       await serve({
-        directory: directory || './site',
+        directory: serveDir,
         port: options.port,
         host: options.host,
         open: options.open,
@@ -269,6 +283,14 @@ export function run() {
 }
 
 async function capture(url, options) {
+  // Compute output directory based on URL domain if not specified
+  let outputDir = options.output;
+  if (!outputDir) {
+    const domain = getDomainFromUrl(url);
+    outputDir = getSiteDir(domain);
+    await ensureSmippoHome();
+  }
+
   const spinner = ora({
     text: 'Initializing browser...',
     isSilent: options.quiet,
@@ -276,7 +298,7 @@ async function capture(url, options) {
 
   const crawler = new Crawler({
     url,
-    output: options.output,
+    output: outputDir,
     depth: parseInt(options.depth, 10),
     scope: options.scope,
     stayInDir: options.stayInDir,
@@ -356,7 +378,22 @@ async function capture(url, options) {
     console.log(chalk.yellow(`    Errors:          ${result.stats.errors}`));
   }
   console.log('');
-  console.log(`  Output: ${chalk.underline(options.output)}`);
+  console.log(`  Output: ${chalk.underline(outputDir)}`);
+
+  // Update global manifest with this capture (tracks all sites regardless of location)
+  try {
+    const domain = getDomainFromUrl(url);
+    await addSiteToGlobalManifest({
+      domain,
+      rootUrl: url,
+      outputDir: outputDir,
+      title: result.pages?.[0]?.title || domain,
+      pagesCount: result.stats.pagesCapt,
+      assetsCount: result.stats.assetsCapt,
+    });
+  } catch {
+    // Silently ignore manifest errors
+  }
 }
 
 async function continueCapture(options) {

package/src/server.js

@@ -6,6 +6,7 @@ import chalk from 'chalk';
 import {exec} from 'child_process';
 import * as p from '@clack/prompts';
 import {readManifest} from './manifest.js';
+import {getAllCapturedSites, getSitesDir} from './utils/home.js';
 
 // MIME type mapping
 const MIME_TYPES = {
@@ -256,7 +257,7 @@ async function generateDirectoryListing(dirPath, urlPath, rootDir) {
           : ''
       }
 		</div>
-		
+
 		${
       !isRoot
         ? `
@@ -273,7 +274,7 @@ async function generateDirectoryListing(dirPath, urlPath, rootDir) {
 		</div>`
         : ''
     }
-		
+
 		<div class="listing">
 			<div class="listing-header">
 				${isRoot ? 'Captured Sites' : `Contents of ${urlPath}`}
@@ -317,7 +318,7 @@ async function generateDirectoryListing(dirPath, urlPath, rootDir) {
           : ''
       }
 		</div>
-		
+
 		<div class="footer">
 			Powered by Smippo • Modern Website Copier
 		</div>
@@ -577,41 +578,103 @@ function truncatePath(p, maxLen) {
 }
 
 /**
- * Get captured sites from a smippo output directory
+ * Get captured sites - either from global manifest or a specific directory
+ * @param {string|null} directory - Specific directory to serve, or null for global
  */
 async function getCapturedSites(directory) {
   const sites = [];
-  const smippoDir = path.join(directory, '.smippo');
 
+  // If no directory specified or it's the global sites dir, use global manifest
+  const globalSitesDir = getSitesDir();
+  const isGlobalMode =
+    !directory || path.resolve(directory) === path.resolve(globalSitesDir);
+
+  if (isGlobalMode) {
+    // Use global manifest to get all captured sites
+    const globalSites = await getAllCapturedSites();
+    for (const site of globalSites) {
+      // Find the domain subdirectory within the site path
+      const domainDir = path.join(site.path, site.domain);
+      const indexInDomain = path.join(domainDir, 'index.html');
+      const indexInRoot = path.join(site.path, 'index.html');
+
+      let hasIndex = false;
+
+      if (await fs.pathExists(indexInDomain)) {
+        hasIndex = true;
+      } else if (await fs.pathExists(indexInRoot)) {
+        hasIndex = true;
+      }
+
+      sites.push({
+        domain: site.domain,
+        fullPath: site.path, // Absolute path to serve from
+        domainPath: site.domain, // Domain subdirectory
+        hasIndex,
+        rootUrl: site.rootUrl,
+        title: site.title || site.domain,
+        pagesCount: site.pagesCount || 0,
+        assetsCount: site.assetsCount || 0,
+        lastUpdated: site.updated || null,
+      });
+    }
+    return sites;
+  }
+
+  // Specific directory mode: check for .smippo directory
+  const smippoDir = path.join(directory, '.smippo');
   if (!(await fs.pathExists(smippoDir))) {
+    // Check if directory itself is a site directory (has index.html)
+    const indexPath = path.join(directory, 'index.html');
+    if (await fs.pathExists(indexPath)) {
+      const dirName = path.basename(directory);
+      sites.push({
+        domain: dirName,
+        fullPath: directory,
+        domainPath: '',
+        hasIndex: true,
+        rootUrl: null,
+        title: dirName,
+        pagesCount: 0,
+        assetsCount: 0,
+        lastUpdated: null,
+      });
+    }
     return sites;
   }
 
-  // Read manifest for site info
+  // Read local manifest for site info
   const manifest = await readManifest(directory);
 
-  // Find domain directories
-  const entries = await fs.readdir(directory);
-  for (const entry of entries) {
-    if (entry.startsWith('.')) continue;
-    const entryPath = path.join(directory, entry);
-    const stat = await fs.stat(entryPath);
-    if (stat.isDirectory()) {
-      // Check if this directory has an index.html
-      const indexPath = path.join(entryPath, 'index.html');
+  if (!manifest?.rootUrl) {
+    return sites;
+  }
+
+  // Extract domain from rootUrl
+  try {
+    const url = new URL(manifest.rootUrl);
+    const mainDomain = url.hostname;
+
+    // Check if the main domain directory exists
+    const domainPath = path.join(directory, mainDomain);
+    if (await fs.pathExists(domainPath)) {
+      const indexPath = path.join(domainPath, 'index.html');
       const hasIndex = await fs.pathExists(indexPath);
 
       sites.push({
-        domain: entry,
-        path: entry,
+        domain: mainDomain,
+        fullPath: directory,
+        domainPath: mainDomain,
         hasIndex,
-        rootUrl: manifest?.rootUrl || null,
-        title: manifest?.pages?.[0]?.title || entry,
-        pagesCount: manifest?.stats?.pagesCapt || 0,
-        assetsCount: manifest?.stats?.assetsCapt || 0,
-        lastUpdated: manifest?.updated || null,
+        rootUrl: manifest.rootUrl,
+        title: manifest.pages?.[0]?.title || mainDomain,
+        pagesCount: manifest.stats?.pagesCapt || 0,
+        assetsCount: manifest.stats?.assetsCapt || 0,
+        lastUpdated: manifest.updated || null,
       });
     }
+  } catch {
+    // Invalid URL in manifest, fall back to directory scan
   }
 
   return sites;
@@ -622,12 +685,15 @@ async function getCapturedSites(directory) {
  */
 export async function serve(options) {
   try {
-    const directory = options.output || options.directory || './site';
-    const resolvedDir = path.resolve(directory);
+    const directory = options.output || options.directory;
+    const globalSitesDir = getSitesDir();
+
+    // Get captured sites (from global manifest if no directory specified)
+    const sites = await getCapturedSites(directory);
 
-    // Check if this is a smippo capture directory
-    const sites = await getCapturedSites(resolvedDir);
-    let sitePath = null;
+    let serveDir = directory ? path.resolve(directory) : null;
+    let openPath = null;
+    let selectedSite = null;
 
     if (sites.length > 0 && process.stdin.isTTY && !options.quiet) {
       // Show interactive site selection
@@ -636,22 +702,30 @@ export async function serve(options) {
 
       if (sites.length === 1) {
         // Single site - auto-select but show info
-        const site = sites[0];
+        selectedSite = sites[0];
         console.log(
-          chalk.dim('  Found captured site: ') + chalk.bold(site.domain),
+          chalk.dim('  Found captured site: ') +
+            chalk.bold(selectedSite.domain),
         );
-        if (site.pagesCount > 0) {
+        if (selectedSite.pagesCount > 0) {
           console.log(
-            chalk.dim(`  ${site.pagesCount} pages, ${site.assetsCount} assets`),
+            chalk.dim(
+              `  ${selectedSite.pagesCount} pages, ${selectedSite.assetsCount} assets`,
+            ),
           );
         }
-        sitePath = site.path;
+        if (selectedSite.fullPath) {
+          console.log(chalk.dim(`  Location: ${selectedSite.fullPath}`));
+        }
       } else {
         // Multiple sites - let user choose
         const siteOptions = sites.map(site => ({
-          value: site.path,
+          value: site,
           label: site.domain,
-          hint: site.pagesCount > 0 ? `${site.pagesCount} pages` : undefined,
+          hint:
+            site.pagesCount > 0
+              ? `${site.pagesCount} pages - ${site.fullPath}`
+              : site.fullPath,
         }));
 
         const selected = await p.select({
@@ -664,20 +738,34 @@ export async function serve(options) {
           process.exit(0);
         }
 
-        sitePath = selected;
+        selectedSite = selected;
       }
       console.log('');
     } else if (sites.length === 1) {
       // Non-interactive mode with single site
-      sitePath = sites[0].path;
+      selectedSite = sites[0];
+    } else if (sites.length === 0 && !directory) {
+      console.log(chalk.yellow('No captured sites found.'));
+      console.log(
+        chalk.dim('  Capture a site first: ') + chalk.cyan('smippo <url>'),
+      );
+      process.exit(0);
+    }
+
+    // Determine serve directory and open path
+    if (selectedSite) {
+      serveDir = selectedSite.fullPath;
+      openPath = selectedSite.domainPath || null;
+    } else if (!serveDir) {
+      serveDir = globalSitesDir;
     }
 
     const serverInfo = await createServer({
-      directory: resolvedDir,
+      directory: serveDir,
       port: options.port || 8080,
       host: options.host || '127.0.0.1',
       open: options.open,
-      openPath: sitePath, // Pass the site path to open
+      openPath: openPath,
       cors: options.cors !== false,
       verbose: options.verbose,
       quiet: options.quiet,

package/src/utils/home.js

@@ -0,0 +1,175 @@
+// @flow
+import fs from 'fs-extra';
+import path from 'path';
+import os from 'os';
+
+const SMIPPO_HOME_DIR = '.smippo';
+const SITES_DIR = 'sites';
+const GLOBAL_MANIFEST_FILE = 'manifest.json';
+
+/**
+ * Get the global smippo home directory (~/.smippo/)
+ */
+export function getSmippoHome() {
+  return path.join(os.homedir(), SMIPPO_HOME_DIR);
+}
+
+/**
+ * Get the sites directory (~/.smippo/sites/)
+ */
+export function getSitesDir() {
+  return path.join(getSmippoHome(), SITES_DIR);
+}
+
+/**
+ * Get the output directory for a specific domain
+ * @param {string} domain - The domain name (e.g., 'example.com')
+ * @returns {string} The full path to the site directory
+ */
+export function getSiteDir(domain) {
+  return path.join(getSitesDir(), domain);
+}
+
+/**
+ * Extract domain from a URL
+ * @param {string} url - The URL to extract domain from
+ * @returns {string} The domain (hostname)
+ */
+export function getDomainFromUrl(url) {
+  try {
+    const parsed = new URL(url);
+    return parsed.hostname;
+  } catch {
+    // If URL parsing fails, return the original string
+    return url;
+  }
+}
+
+/**
+ * Ensure the smippo home directory exists
+ */
+export async function ensureSmippoHome() {
+  const homeDir = getSmippoHome();
+  const sitesDir = getSitesDir();
+
+  await fs.ensureDir(homeDir);
+  await fs.ensureDir(sitesDir);
+
+  return homeDir;
+}
+
+/**
+ * Get the global manifest path (~/.smippo/manifest.json)
+ */
+export function getGlobalManifestPath() {
+  return path.join(getSmippoHome(), GLOBAL_MANIFEST_FILE);
+}
+
+/**
+ * Read the global manifest (list of all captured sites)
+ */
+export async function readGlobalManifest() {
+  const manifestPath = getGlobalManifestPath();
+
+  if (!(await fs.pathExists(manifestPath))) {
+    return {
+      version: '1.0.0',
+      sites: [],
+    };
+  }
+
+  try {
+    const content = await fs.readFile(manifestPath, 'utf8');
+    return JSON.parse(content);
+  } catch {
+    return {
+      version: '1.0.0',
+      sites: [],
+    };
+  }
+}
+
+/**
+ * Write the global manifest
+ */
+export async function writeGlobalManifest(manifest) {
+  await ensureSmippoHome();
+  const manifestPath = getGlobalManifestPath();
+  await fs.writeFile(manifestPath, JSON.stringify(manifest, null, 2), 'utf8');
+}
+
+/**
+ * Add a site to the global manifest
+ * @param {Object} siteInfo - Site information
+ * @param {string} siteInfo.domain - Domain name
+ * @param {string} siteInfo.rootUrl - Original URL
+ * @param {string} siteInfo.outputDir - Directory where site was saved
+ * @param {string} [siteInfo.title] - Page title
+ * @param {number} [siteInfo.pagesCount] - Number of pages captured
+ * @param {number} [siteInfo.assetsCount] - Number of assets captured
+ */
+export async function addSiteToGlobalManifest(siteInfo) {
+  const manifest = await readGlobalManifest();
+
+  // Use outputDir as the unique key (same domain can be saved to different dirs)
+  const outputPath = path.resolve(siteInfo.outputDir);
+  const existingIndex = manifest.sites.findIndex(s => s.path === outputPath);
+
+  const siteEntry = {
+    domain: siteInfo.domain,
+    rootUrl: siteInfo.rootUrl,
+    title: siteInfo.title || siteInfo.domain,
+    path: outputPath,
+    created: siteInfo.created || new Date().toISOString(),
+    updated: new Date().toISOString(),
+    pagesCount: siteInfo.pagesCount || 0,
+    assetsCount: siteInfo.assetsCount || 0,
+  };
+
+  if (existingIndex >= 0) {
+    // Update existing entry (preserve created date)
+    manifest.sites[existingIndex] = {
+      ...siteEntry,
+      created: manifest.sites[existingIndex].created,
+    };
+  } else {
+    // Add new entry
+    manifest.sites.push(siteEntry);
+  }
+
+  await writeGlobalManifest(manifest);
+  return manifest;
+}
+
+/**
+ * Remove a site from the global manifest
+ */
+export async function removeSiteFromGlobalManifest(domain) {
+  const manifest = await readGlobalManifest();
+  manifest.sites = manifest.sites.filter(s => s.domain !== domain);
+  await writeGlobalManifest(manifest);
+  return manifest;
+}
+
+/**
+ * Get all captured sites from the global manifest
+ */
+export async function getAllCapturedSites() {
+  const manifest = await readGlobalManifest();
+
+  // Verify each site still exists
+  const validSites = [];
+  for (const site of manifest.sites) {
+    if (await fs.pathExists(site.path)) {
+      validSites.push(site);
+    }
+  }
+
+  // Update manifest if some sites were removed
+  if (validSites.length !== manifest.sites.length) {
+    manifest.sites = validSites;
+    await writeGlobalManifest(manifest);
+  }
+
+  return validSites;
+}