@d-zero/replicator 0.5.4 → 0.6.1

package/README.md

@@ -1,6 +1,6 @@
 # `@d-zero/replicator`
 
-ウェブページとそのリソースをレスポンシブ画像対応でローカルディレクトリに複製するツールです。
+ウェブページとそのリソースをレスポンシブ画像対応でローカルディレクトリに複製するツールです。複数のURLを並列処理し、効率的にリソースを取得します。
 
 ## インストール
 
@@ -13,7 +13,7 @@ npm install @d-zero/replicator
 ### CLI
 
 ```bash
-npx @d-zero/replicator <url> -o <output-directory> [options]
+npx @d-zero/replicator <url...> -o <output-directory> [options]
 ```
 
 #### オプション
@@ -21,8 +21,16 @@ npx @d-zero/replicator <url> -o <output-directory> [options]
 - `-o, --output <dir>`: 出力ディレクトリ（必須）
 - `-t, --timeout <ms>`: リクエストタイムアウト（ミリ秒、デフォルト: 30000）
 - `-d, --devices <devices>`: デバイスプリセット（カンマ区切り、デフォルト: desktop-compact,mobile）
+- `-l, --limit <number>`: 並列処理数の上限（デフォルト: 3）
+- `--only <type>`: ダウンロード対象を限定（`page` または `resource`）
 - `-v, --verbose`: 詳細ログモード
 
+##### `--only` オプション
+
+- `page`: HTMLページのみをダウンロード（リソーススキャンをスキップして高速化）
+- `resource`: リソース（CSS、JS、画像など）のみをダウンロード（HTMLページを除外）
+- 未指定: すべてのファイルをダウンロード（デフォルト動作）
+
 #### 利用可能なデバイスプリセット
 
 - `desktop`: 1400px幅
@@ -36,14 +44,26 @@ npx @d-zero/replicator <url> -o <output-directory> [options]
 #### 使用例
 
 ```bash
-# デフォルトデバイス（desktop-compact, mobile）
+# 単一URL（デフォルトデバイス: desktop-compact, mobile）
 npx @d-zero/replicator https://example.com -o ./output
 
+# 複数URLを並列処理
+npx @d-zero/replicator https://example.com/page1 https://example.com/page2 -o ./output
+
+# 並列数を制限
+npx @d-zero/replicator https://example.com/page1 https://example.com/page2 -o ./output --limit 2
+
 # カスタムデバイス指定
 npx @d-zero/replicator https://example.com -o ./output --devices desktop,tablet,mobile
 
 # タイムアウト指定
 npx @d-zero/replicator https://example.com -o ./output --timeout 60000
+
+# HTMLページのみダウンロード（高速）
+npx @d-zero/replicator https://example.com -o ./output --only page
+
+# リソースのみダウンロード（HTMLを除外）
+npx @d-zero/replicator https://example.com -o ./output --only resource
 ```
 
 ### プログラマティック使用
@@ -51,11 +71,27 @@ npx @d-zero/replicator https://example.com -o ./output --timeout 60000
 ```typescript
 import { replicate } from '@d-zero/replicator';
 
-// デフォルトデバイス
-await replicate('https://example.com', './output');
+// 単一URL
+await replicate({
+	urls: ['https://example.com'],
+	outputDir: './output',
+});
+
+// 複数URLを並列処理
+await replicate({
+	urls: [
+		'https://example.com/page1',
+		'https://example.com/page2',
+		'https://example.com/page3',
+	],
+	outputDir: './output',
+	limit: 2, // 最大2つのURLを同時処理
+});
 
 // カスタムデバイス
-await replicate('https://example.com', './output', {
+await replicate({
+	urls: ['https://example.com'],
+	outputDir: './output',
 	devices: {
 		desktop: { width: 1400 },
 		mobile: { width: 375, resolution: 2 },
@@ -63,17 +99,34 @@ await replicate('https://example.com', './output', {
 	timeout: 30000,
 	verbose: true,
 });
+
+// HTMLページのみダウンロード
+await replicate({
+	urls: ['https://example.com'],
+	outputDir: './output',
+	only: 'page',
+});
+
+// リソースのみダウンロード
+await replicate({
+	urls: ['https://example.com'],
+	outputDir: './output',
+	only: 'resource',
+});
 ```
 
 ## 機能
 
+- **並列処理**: 複数のURLを並列で効率的に処理
+- **メモリ効率**: リソースを直接ディスクに保存してメモリ使用量を最小化
+- **選択的ダウンロード**: `--only`オプションでHTMLページのみまたはリソースのみをダウンロード可能
 - **レスポンシブ画像対応**: 複数のデバイス幅で`<picture>`要素やメディアクエリのリソースを取得
 - **遅延読み込み対応**: ページを自動スクロールして`loading=lazy`や`IntersectionObserver`ベースのコンテンツを取得
 - **マルチデバイスシミュレーション**: 様々なデバイス幅と解像度をシミュレートして包括的なリソース取得を実現
 - HTMLページのディレクトリ構造を保持してダウンロード
 - 関連するすべてのリソース（CSS、JS、画像など）を取得
 - リソース間の相対リンクを維持
-- 同一ホストのリソースのみサポート
+- 同一ホストのリソースのみサポート（複数ホストが検出された場合はエラー）
 - 元のファイル拡張子とパスを保持
 
 ## License

package/dist/child-process.d.ts

@@ -0,0 +1 @@
+export {};

package/dist/child-process.js

@@ -0,0 +1,94 @@
+import { createChildProcess } from '@d-zero/puppeteer-dealer';
+import { beforePageScan, devicePresets } from '@d-zero/puppeteer-page-scan';
+import { scrollAllOver } from '@d-zero/puppeteer-scroll';
+/**
+ * Add resource path to the set with MIME type encoding if needed
+ * @param pathname - Resource pathname
+ * @param mimeType - MIME type from response headers (optional)
+ * @returns Encoded resource path
+ */
+function encodeResourcePath(pathname, mimeType) {
+    // Normalize empty pathname to "/"
+    if (pathname === '') {
+        pathname = '/';
+    }
+    // Check if the last segment has an extension
+    const lastSlashIndex = pathname.lastIndexOf('/');
+    const lastSegment = lastSlashIndex === -1 ? pathname : pathname.slice(lastSlashIndex + 1);
+    const hasExtension = lastSegment.includes('.');
+    // For paths without extension, encode with MIME type if available
+    if (!hasExtension && mimeType) {
+        return `${pathname}:::${mimeType}`;
+    }
+    // For paths with extension or without MIME type, return as-is
+    return pathname;
+}
+createChildProcess((param) => {
+    const { devices, timeout } = param;
+    return {
+        async eachPage({ page, url }, logger) {
+            const resourcePaths = new Set();
+            const pageHostname = new URL(url).hostname;
+            // Add the page URL itself first (in case response event is missed)
+            const pageUrlObj = new URL(url);
+            const pagePathname = pageUrlObj.pathname;
+            resourcePaths.add(encodeResourcePath(pagePathname, 'text/html'));
+            // Listen to all network responses
+            const responseHandler = (response) => {
+                const responseUrl = response.url();
+                // Skip data URLs
+                if (responseUrl.startsWith('data:')) {
+                    return;
+                }
+                // Skip non-GET requests (POST, PUT, etc. cannot be replicated)
+                if (response.request().method() !== 'GET') {
+                    return;
+                }
+                // Skip non-successful responses (not 2xx)
+                if (response.status() < 200 || response.status() >= 300) {
+                    return;
+                }
+                const resourceUrlObj = new URL(responseUrl);
+                // Skip different domain resources
+                if (resourceUrlObj.hostname !== pageHostname) {
+                    return;
+                }
+                // Get pathname and MIME type
+                const resourcePath = resourceUrlObj.pathname;
+                const contentType = response.headers()['content-type'];
+                const mimeType = contentType?.split(';')[0]?.trim();
+                // Add resource with MIME encoding if needed
+                resourcePaths.add(encodeResourcePath(resourcePath, mimeType));
+            };
+            page.on('response', responseHandler);
+            const defaultSizes = {
+                'desktop-compact': devicePresets['desktop-compact'],
+                mobile: devicePresets.mobile,
+            };
+            const targetSizes = devices ?? defaultSizes;
+            // Scan the page across all device sizes
+            for (const [sizeName, size] of Object.entries(targetSizes)) {
+                logger(`📱 Scanning with ${sizeName} (${size.width}px)`);
+                await beforePageScan(page, url, {
+                    name: sizeName,
+                    width: size.width,
+                    timeout,
+                }).catch((error) => {
+                    logger(`❌ Failed to scan ${sizeName}: ${error.message}`);
+                    throw error;
+                });
+                // Scroll to load lazy resources
+                await scrollAllOver(page).catch((error) => {
+                    logger(`❌ Failed to scroll ${sizeName}: ${error.message}`);
+                    throw error;
+                });
+            }
+            page.off('response', responseHandler);
+            logger(`📦 Collected ${resourcePaths.size} resources`);
+            return {
+                url,
+                encodedUrls: [...resourcePaths],
+            };
+        },
+    };
+});

package/dist/cli.d.ts

@@ -0,0 +1,2 @@
+#!/usr/bin/env node
+export {};

package/dist/cli.js

@@ -8,14 +8,17 @@ const { options, args } = createCLI({
         v: 'verbose',
         t: 'timeout',
         d: 'devices',
+        l: 'limit',
     },
     usage: [
-        'Usage: replicator <url> -o <output-directory> [options]',
+        'Usage: replicator <url1> [url2...] -o <output-directory> [options]',
         '',
         'Options:',
         '  -o, --output <dir>        Output directory (required)',
         '  -t, --timeout <ms>        Request timeout in milliseconds (default: 30000)',
         '  -d, --devices <devices>   Device presets (comma-separated, default: desktop-compact,mobile)',
+        '  -l, --limit <number>      Parallel execution limit (default: 3)',
+        '  --only <type>             Download only specified type: page or resource',
         '  -v, --verbose             Enable verbose logging',
         '',
         'Available device presets:',
@@ -23,36 +26,46 @@ const { options, args } = createCLI({
         '',
         'Examples:',
         '  replicator https://example.com -o ./output',
+        '  replicator https://example.com/page1 https://example.com/page2 -o ./output',
         '  replicator https://example.com -o ./output --devices desktop,tablet',
-        '  replicator https://example.com -o ./output --timeout 60000',
+        '  replicator https://example.com -o ./output --timeout 60000 --limit 5',
+        '  replicator https://example.com -o ./output --only page',
+        '  replicator https://example.com -o ./output --only resource',
     ],
     parseArgs: (cli) => ({
         ...parseCommonOptions(cli),
         output: cli.output,
         timeout: cli.timeout ? Number(cli.timeout) : undefined,
         devices: cli.devices,
+        limit: cli.limit ? Number(cli.limit) : undefined,
+        only: cli.only,
     }),
     validateArgs: (options, cli) => {
+        if (options.only && options.only !== 'page' && options.only !== 'resource') {
+            return false;
+        }
         return !!(cli._.length > 0 && options.output);
     },
 });
-const url = args[0];
+const urls = args.filter((arg) => typeof arg === 'string');
 const outputDir = options.output;
-if (!url || typeof url !== 'string') {
+if (urls.length === 0) {
     // eslint-disable-next-line no-console
-    console.error('❌ Error: URL is required');
+    console.error('❌ Error: At least one URL is required');
     process.exit(1);
 }
 try {
     const deviceNames = options.devices ? parseList(options.devices) : undefined;
     const devices = parseDevicesOption(deviceNames);
-    await replicate(url, outputDir, {
+    await replicate({
+        urls,
+        outputDir,
         verbose: options.verbose ?? false,
         timeout: options.timeout,
         devices,
+        limit: options.limit,
+        only: options.only,
     });
-    // eslint-disable-next-line no-console
-    console.log(`✅ Successfully replicated ${url} to ${outputDir}`);
 }
 catch (error) {
     if (error instanceof Error) {

package/dist/index.d.ts

@@ -0,0 +1,26 @@
+import type { ReplicateOptions } from './types.js';
+/**
+ * Replicate web pages with all their resources to local directories
+ *
+ * ## Architecture
+ *
+ * This implementation uses a two-phase architecture for memory efficiency:
+ *
+ * ### Phase 1: Metadata Collection
+ * - Each URL is processed in a separate child process using puppeteer-dealer
+ * - Child processes scan pages with Puppeteer and collect resource URLs
+ * - For URLs ending with '/' (e.g., https://example.com/), MIME type is captured
+ * and encoded as "url:::MIME/type" format
+ * - Only metadata (URLs + MIME types) is returned to parent - no buffer data
+ *
+ * ### Phase 2: Resource Download
+ * - Parent process aggregates all metadata and removes duplicates
+ * - Parses encoded URLs to determine correct local paths
+ * - Downloads resources via fetch() and immediately writes to disk
+ * - No resource content is kept in memory
+ *
+ * This approach minimizes memory usage by avoiding duplicate I/O operations
+ * and keeping buffer data out of inter-process communication.
+ * @param options - Replication options
+ */
+export declare function replicate(options: ReplicateOptions): Promise<void>;

package/dist/index.js

@@ -1,287 +1,175 @@
-import { promises as fs } from 'node:fs';
 import path from 'node:path';
-import { URL } from 'node:url';
-import { beforePageScan, devicePresets } from '@d-zero/puppeteer-page-scan';
-import { launch } from 'puppeteer';
+import { deal, createProcess } from '@d-zero/puppeteer-dealer';
+import { devicePresets } from '@d-zero/puppeteer-page-scan';
+import { validateSameHost } from '@d-zero/shared/validate-same-host';
+import c from 'ansi-colors';
+import { downloadResources } from './resource-downloader.js';
 /**
- *
- * @param url
- * @param outputDir
- * @param options
+ * Encode resource path with MIME type if needed
+ * @param pathname - Resource pathname
+ * @param mimeType - MIME type (optional)
+ * @returns Encoded resource path
  */
-export async function replicate(url, outputDir, options = {}) {
-    const { verbose = false, timeout = 30_000, devices } = options;
-    const defaultSizes = {
-        'desktop-compact': devicePresets['desktop-compact'],
-        mobile: devicePresets.mobile,
-    };
-    const targetSizes = devices ?? defaultSizes;
-    const log = (message) => {
-        if (verbose) {
-            // eslint-disable-next-line no-console
-            console.log(message);
-        }
-    };
-    // Always show these key progress messages
-    const progress = (message) => {
-        // eslint-disable-next-line no-console
-        console.log(message);
-    };
-    const baseUrl = new URL(url);
-    const allResources = [];
-    progress(`🚀 Starting replication of ${url}`);
-    log(`   Output directory: ${outputDir}`);
-    log(`   Device sizes: ${Object.keys(targetSizes).join(', ')}`);
-    progress(`🌐 Launching browser...`);
-    const browser = await launch({
-        headless: true,
-    });
-    try {
-        // Process each device size
-        for (const [sizeName, sizeConfig] of Object.entries(targetSizes)) {
-            const { width } = sizeConfig;
-            const resolution = 'resolution' in sizeConfig ? sizeConfig.resolution : undefined;
-            progress(`📱 Processing ${sizeName} (${width}px${resolution ? `, ${resolution}x` : ''})...`);
-            const page = await browser.newPage();
-            const sizeResources = [];
-            try {
-                await processPageForSize(page, url, baseUrl, sizeResources, {
-                    sizeName,
-                    width,
-                    resolution,
-                    timeout,
-                    log,
-                    progress,
-                });
-                // Merge resources, avoiding duplicates
-                for (const resource of sizeResources) {
-                    const existing = allResources.find((r) => r.url === resource.url);
-                    if (!existing) {
-                        allResources.push(resource);
-                    }
-                    else if (!existing.content && resource.content) {
-                        existing.content = resource.content;
-                    }
-                }
-            }
-            finally {
-                await page.close().catch((error) => {
-                    log(`⚠️ Warning: Failed to close page for ${sizeName}: ${error instanceof Error ? error.message : String(error)}`);
-                });
-            }
-        }
-        const resourceCount = allResources.length;
-        const downloadedCount = allResources.filter((r) => r.content).length;
-        progress(`📄 Found ${resourceCount} total resources (${downloadedCount} downloaded successfully)`);
-        // Ensure output directory exists
-        progress(`📁 Creating output directory...`);
-        await fs.mkdir(outputDir, { recursive: true });
-        // Save all resources
-        progress(`💾 Saving files to disk...`);
-        const savedCount = await saveResources(allResources, outputDir, log, progress);
-        progress(`🎉 Replication complete! ${savedCount} files saved to ${outputDir}`);
+function encodeResourcePath(pathname, mimeType) {
+    // Normalize empty pathname to "/"
+    if (pathname === '') {
+        pathname = '/';
     }
-    finally {
-        progress(`🔧 Cleaning up browser...`);
-        await browser.close().catch((error) => {
-            // Log browser close errors but don't throw them
-            log(`⚠️ Warning: Failed to close browser: ${error instanceof Error ? error.message : String(error)}`);
-        });
+    // Check if the last segment has an extension
+    const lastSlashIndex = pathname.lastIndexOf('/');
+    const lastSegment = lastSlashIndex === -1 ? pathname : pathname.slice(lastSlashIndex + 1);
+    const hasExtension = lastSegment.includes('.');
+    // For paths without extension, encode with MIME type if available
+    if (!hasExtension && mimeType) {
+        return `${pathname}:::${mimeType}`;
     }
+    // For paths with extension or without MIME type, return as-is
+    return pathname;
 }
 /**
- * Process a page for a specific device size
- * @param page
- * @param url
- * @param baseUrl
- * @param resources
- * @param options
- * @param options.sizeName
- * @param options.width
- * @param options.resolution
- * @param options.timeout
- * @param options.log
- * @param options.progress
+ * Collect page URLs without resource scanning (page-only mode)
+ * @param urls - Array of URLs to process
+ * @param progress - Progress logger function
+ * @returns Set of encoded URLs
  */
-async function processPageForSize(page, url, baseUrl, resources, options) {
-    const { sizeName, width, resolution, timeout, log, progress } = options;
-    const requestPromises = [];
-    // Set up resource detection
-    page.on('request', (request) => {
-        const requestUrl = request.url();
-        const requestUrlObj = new URL(requestUrl);
-        // Only handle same-host resources
-        if (requestUrlObj.hostname === baseUrl.hostname) {
-            log(`📥 [${sizeName}] Intercepting: ${requestUrl}`);
-            const localPath = urlToLocalPath(requestUrl);
-            const resourceType = getResourceType(requestUrl);
-            // Check if this resource is already tracked
-            if (!resources.some((r) => r.url === requestUrl)) {
-                resources.push({
-                    url: requestUrl,
-                    localPath,
-                    type: resourceType,
-                });
-            }
-        }
-        else {
-            log(`🚫 [${sizeName}] Skipping external resource: ${requestUrl}`);
-        }
-    });
-    page.on('response', (response) => {
-        const responseUrl = response.url();
-        const responseUrlObj = new URL(responseUrl);
-        // Only handle same-host resources
-        if (responseUrlObj.hostname === baseUrl.hostname) {
-            const promise = (async () => {
-                const resource = resources.find((r) => r.url === responseUrl);
-                if (resource && response.ok()) {
-                    await response
-                        .buffer()
-                        .then((buffer) => {
-                        resource.content = buffer;
-                        log(`✅ [${sizeName}] Downloaded: ${responseUrl}`);
-                    })
-                        .catch((error) => {
-                        const errorMessage = error instanceof Error ? error.message : String(error);
-                        log(`❌ [${sizeName}] Failed to download: ${responseUrl} - ${errorMessage}`);
-                    });
-                }
-                else if (resource) {
-                    log(`❌ [${sizeName}] Resource failed (${response.status()}): ${responseUrl}`);
-                }
-            })();
-            requestPromises.push(promise);
-        }
-    });
-    // Set viewport and navigate using beforePageScan (which includes scrolling)
-    progress(`📡 [${sizeName}] Setting viewport and navigating...`);
-    await beforePageScan(page, url, {
-        name: sizeName,
-        width,
-        resolution,
-        timeout,
-        listener: (phase, data) => {
-            switch (phase) {
-                case 'setViewport': {
-                    const setViewportData = data;
-                    log(`📱 [${sizeName}] Viewport set: ${setViewportData.width}px${setViewportData.resolution ? ` @ ${setViewportData.resolution}x` : ''}`);
-                    break;
-                }
-                case 'load': {
-                    const loadData = data;
-                    log(`📄 [${sizeName}] Page loaded (${loadData.type})`);
-                    break;
-                }
-                case 'scroll': {
-                    const scrollData = data;
-                    switch (scrollData.message) {
-                        case 'Start scrolling': {
-                            log(`📜 [${sizeName}] Starting scroll to trigger lazy loading...`);
-                            break;
-                        }
-                        case 'End of page': {
-                            log(`📜 [${sizeName}] Scroll completed (${scrollData.scrollY}/${scrollData.scrollHeight}px)`);
-                            break;
-                        }
-                        case 'Scrolling': {
-                            const progress = Math.round((scrollData.scrollY / scrollData.scrollHeight) * 100);
-                            log(`📜 [${sizeName}] Scrolling progress: ${progress}% (${scrollData.scrollY}/${scrollData.scrollHeight}px)`);
-                            break;
-                        }
-                        // No default
-                    }
-                    break;
-                }
-            }
-        },
-    });
-    progress(`⏳ [${sizeName}] Waiting for all resources to load...`);
-    // Wait for all downloads to complete
-    await Promise.all(requestPromises);
-    const resourceCount = resources.length;
-    const downloadedCount = resources.filter((r) => r.content).length;
-    progress(`📄 [${sizeName}] Found ${resourceCount} resources (${downloadedCount} downloaded)`);
+function collectPageUrlsOnly(urls, progress) {
+    progress(c.bold.yellow('📄 Page-only mode: Skipping resource collection...'));
+    progress('');
+    const encodedUrls = new Set();
+    for (const url of urls) {
+        const urlObj = new URL(url);
+        const pathname = urlObj.pathname || '/';
+        // Encode as HTML page
+        const encodedPath = encodeResourcePath(pathname, 'text/html');
+        encodedUrls.add(encodedPath);
+    }
+    progress(c.bold.green(`✅ Prepared ${encodedUrls.size} page(s) for download`));
+    return encodedUrls;
 }
 /**
- *
- * @param resources
- * @param outputDir
- * @param log
- * @param progress
+ * Collect all resource URLs using Puppeteer scanning
+ * @param urls - Array of URLs to process
+ * @param targetSizes - Device sizes for responsive scanning
+ * @param timeout - Request timeout
+ * @param verbose - Enable verbose logging
+ * @param limit - Parallel execution limit
+ * @param progress - Progress logger function
+ * @returns Set of encoded URLs
  */
-async function saveResources(resources, outputDir, log, progress) {
-    let savedCount = 0;
-    const totalResources = resources.filter((r) => r.content).length;
-    for (const resource of resources) {
-        if (resource.content) {
-            const fullPath = path.join(outputDir, resource.localPath);
-            const dir = path.dirname(fullPath);
-            try {
-                await fs.mkdir(dir, { recursive: true });
-                await fs.writeFile(fullPath, resource.content);
-                savedCount++;
-                // Show progress every 10 files or for the last file
-                if (savedCount % 10 === 0 || savedCount === totalResources) {
-                    progress(`   Saved ${savedCount}/${totalResources} files...`);
-                }
-                log(`💾 Saved: ${resource.localPath}`);
-            }
-            catch (error) {
-                const errorMessage = error instanceof Error ? error.message : String(error);
-                log(`❌ Failed to save ${resource.localPath}: ${errorMessage}`);
-                progress(`   ⚠️  Failed to save ${resource.localPath}`);
-                // Continue with other resources instead of failing completely
+async function collectAllResourceUrls(urls, targetSizes, timeout, verbose, limit, progress) {
+    progress(c.bold.yellow('📡 Phase 1: Collecting resource metadata...'));
+    const results = [];
+    await deal(urls.map((url) => ({ id: null, url })), (_, done, total) => {
+        const percentage = Math.round((done / total) * 100);
+        return `${c.bold.cyan('🌐 Replicating')} ${done}/${total} (${percentage}%)`;
+    }, () => createProcess(path.resolve(import.meta.dirname, 'child-process.js'), {
+        devices: targetSizes,
+        timeout,
+    }, {}), {
+        verbose,
+        limit,
+        each: (result) => {
+            results.push(result);
+        },
+    });
+    progress('');
+    progress(c.bold.green(`✅ Phase 1 complete: Collected metadata from ${results.length} URL(s)`));
+    // Log collected URLs in verbose mode
+    if (verbose) {
+        progress('');
+        progress(c.gray('📋 Collected URLs by page:'));
+        for (const result of results) {
+            progress(c.gray(`   ${result.url}:`));
+            for (const encodedUrl of result.encodedUrls) {
+                progress(c.gray(`     - ${encodedUrl}`));
             }
         }
     }
-    return savedCount;
-}
-/**
- *
- * @param url
- */
-function urlToLocalPath(url) {
-    const urlObj = new URL(url);
-    let pathname = urlObj.pathname;
-    // Remove leading slash
-    if (pathname.startsWith('/')) {
-        pathname = pathname.slice(1);
-    }
-    // If path is empty or ends with /, treat as index.html
-    if (pathname === '' || pathname.endsWith('/')) {
-        pathname = pathname + 'index.html';
-    }
-    // If no extension, add .html
-    if (!pathname.includes('.')) {
-        pathname = pathname + '.html';
+    // Aggregate all resource URLs
+    const encodedUrls = new Set();
+    for (const result of results) {
+        for (const encodedUrl of result.encodedUrls) {
+            encodedUrls.add(encodedUrl);
+        }
     }
-    return pathname;
+    return encodedUrls;
 }
 /**
+ * Replicate web pages with all their resources to local directories
  *
- * @param url
+ * ## Architecture
+ *
+ * This implementation uses a two-phase architecture for memory efficiency:
+ *
+ * ### Phase 1: Metadata Collection
+ * - Each URL is processed in a separate child process using puppeteer-dealer
+ * - Child processes scan pages with Puppeteer and collect resource URLs
+ * - For URLs ending with '/' (e.g., https://example.com/), MIME type is captured
+ * and encoded as "url:::MIME/type" format
+ * - Only metadata (URLs + MIME types) is returned to parent - no buffer data
+ *
+ * ### Phase 2: Resource Download
+ * - Parent process aggregates all metadata and removes duplicates
+ * - Parses encoded URLs to determine correct local paths
+ * - Downloads resources via fetch() and immediately writes to disk
+ * - No resource content is kept in memory
+ *
+ * This approach minimizes memory usage by avoiding duplicate I/O operations
+ * and keeping buffer data out of inter-process communication.
+ * @param options - Replication options
  */
-function getResourceType(url) {
-    const urlObj = new URL(url);
-    const pathname = urlObj.pathname.toLowerCase();
-    if (pathname.endsWith('.html') ||
-        pathname.endsWith('.htm') ||
-        pathname === '/' ||
-        !pathname.includes('.')) {
-        return 'html';
+export async function replicate(options) {
+    const { urls, outputDir, verbose = false, timeout = 30_000, devices, limit = 3, only, } = options;
+    if (urls.length === 0) {
+        throw new Error('At least one URL is required');
     }
-    if (pathname.endsWith('.css')) {
-        return 'css';
-    }
-    if (pathname.endsWith('.js') || pathname.endsWith('.mjs')) {
-        return 'js';
-    }
-    if (/\.(?:jpg|jpeg|png|gif|svg|webp|ico)$/.test(pathname)) {
-        return 'image';
-    }
-    if (/\.(?:woff|woff2|ttf|otf|eot)$/.test(pathname)) {
-        return 'font';
+    // Validate that all URLs share the same hostname
+    validateSameHost(urls);
+    const log = (message) => {
+        if (!verbose) {
+            return;
+        }
+        // eslint-disable-next-line no-console
+        console.log(message);
+    };
+    const progress = (message) => {
+        // eslint-disable-next-line no-console
+        console.log(message);
+    };
+    const defaultSizes = {
+        'desktop-compact': devicePresets['desktop-compact'],
+        mobile: devicePresets.mobile,
+    };
+    const targetSizes = devices ?? defaultSizes;
+    progress(c.bold.cyan(`🌐 Replicating ${urls.length} URL(s)`));
+    progress(c.gray(`   Output: ${outputDir}`));
+    progress(c.gray(`   Parallel limit: ${limit}`));
+    progress('');
+    // Phase 1: Collect resource metadata from all URLs
+    let allEncodedUrls;
+    switch (only) {
+        case 'page': {
+            allEncodedUrls = collectPageUrlsOnly(urls, progress);
+            break;
+        }
+        case 'resource':
+        case undefined: {
+            allEncodedUrls = await collectAllResourceUrls(urls, targetSizes, timeout, verbose, limit, progress);
+            break;
+        }
+        default: {
+            throw new Error(`Invalid only option: ${only}`);
+        }
     }
-    return 'other';
+    progress('');
+    // Phase 2: Download resources
+    progress(c.bold.yellow('📦 Phase 2: Downloading resources...'));
+    log(`   Total unique resources: ${allEncodedUrls.size}`);
+    // Use the first URL as base URL for constructing full URLs
+    const baseUrl = urls[0];
+    // Download all resources
+    await downloadResources([...allEncodedUrls], baseUrl, outputDir, progress, verbose, only);
+    progress('');
+    progress(c.bold.green(`✅ Replication complete!`));
+    progress(c.gray(`   All resources saved to: ${outputDir}`));
 }

package/dist/resource-downloader.d.ts

@@ -0,0 +1,10 @@
+/**
+ * Download and save resources to disk
+ * @param encodedPaths - Array of encoded pathnames
+ * @param baseUrl - Base URL to construct full URLs
+ * @param outputDir - Output directory
+ * @param logger - Logger function
+ * @param verbose - Enable verbose output
+ * @param only - Download only specified type: page or resource
+ */
+export declare function downloadResources(encodedPaths: string[], baseUrl: string, outputDir: string, logger: (message: string) => void, verbose?: boolean, only?: 'page' | 'resource'): Promise<void>;

package/dist/resource-downloader.js

@@ -0,0 +1,130 @@
+import { mkdir, writeFile } from 'node:fs/promises';
+import path from 'node:path';
+import { deal } from '@d-zero/dealer';
+import { mimeToExtension } from '@d-zero/shared/mime-to-extension';
+import { urlToLocalPath } from '@d-zero/shared/url-to-local-path';
+import c from 'ansi-colors';
+/**
+ * Parse encoded pathname and return the actual URL and local path
+ * @param encodedPath - pathname or "pathname:::MIME/type" format
+ * @param baseUrl - Base URL to construct full URL from pathname
+ */
+function parseEncodedPath(encodedPath, baseUrl) {
+    const parts = encodedPath.split(':::');
+    if (parts.length === 2) {
+        // Format: "pathname:::MIME/type"
+        const pathname = parts[0];
+        const mimeType = parts[1];
+        const url = new URL(pathname, baseUrl).href;
+        const extension = mimeToExtension(mimeType);
+        const localPath = urlToLocalPath(url, extension);
+        return { url, localPath };
+    }
+    // Regular pathname without MIME encoding
+    const pathname = encodedPath;
+    const url = new URL(pathname, baseUrl).href;
+    const localPath = urlToLocalPath(url, '');
+    return { url, localPath };
+}
+/**
+ * Download and save resources to disk
+ * @param encodedPaths - Array of encoded pathnames
+ * @param baseUrl - Base URL to construct full URLs
+ * @param outputDir - Output directory
+ * @param logger - Logger function
+ * @param verbose - Enable verbose output
+ * @param only - Download only specified type: page or resource
+ */
+export async function downloadResources(encodedPaths, baseUrl, outputDir, logger, verbose = false, only) {
+    const uniqueResources = new Map();
+    // Parse all encoded pathnames
+    for (const encodedPath of encodedPaths) {
+        const { url, localPath } = parseEncodedPath(encodedPath, baseUrl);
+        // Filter based on 'only' option
+        // Note: HTML pages always have .html extension after parseEncodedPath
+        // - either from "pathname:::text/html" encoding via mimeToExtension
+        // - or from original URL path already having .html extension
+        const isHtmlPage = localPath.endsWith('.html');
+        if (only === 'resource' && isHtmlPage) {
+            // Skip HTML pages in resource-only mode
+            continue;
+        }
+        if (only === 'page' && !isHtmlPage) {
+            // Skip non-HTML resources in page-only mode
+            // (though this shouldn't happen if Phase 1 was skipped correctly)
+            continue;
+        }
+        if (!uniqueResources.has(localPath)) {
+            uniqueResources.set(localPath, { url, localPath, encodedPath });
+        }
+    }
+    const tasks = [...uniqueResources.values()];
+    if (tasks.length === 0) {
+        logger(c.yellow('⚠️  No resources to download'));
+        return;
+    }
+    logger(`📥 Downloading ${tasks.length} unique resources...`);
+    logger('');
+    let downloaded = 0;
+    let failed = 0;
+    await deal(tasks, (task, update, index) => {
+        const fileId = index.toString().padStart(4, '0');
+        const lineHeader = `%braille% ${c.bgWhite(` ${fileId} `)} ${c.gray(task.localPath)}: `;
+        return async () => {
+            update(`${lineHeader}Fetching%dots%`);
+            const response = await fetch(task.url).catch((error) => {
+                update(`${lineHeader}${c.red(`❌ Fetch failed: ${error.message}`)}`);
+                failed++;
+                return null;
+            });
+            if (!response) {
+                return;
+            }
+            if (!response.ok) {
+                update(`${lineHeader}${c.red(`❌ HTTP ${response.status}`)}`);
+                failed++;
+                return;
+            }
+            update(`${lineHeader}Reading content%dots%`);
+            const content = Buffer.from(await response.arrayBuffer());
+            const fullPath = path.join(outputDir, task.localPath);
+            const dir = path.dirname(fullPath);
+            update(`${lineHeader}Creating directory%dots%`);
+            const mkdirSuccess = await mkdir(dir, { recursive: true })
+                .then(() => true)
+                .catch((error) => {
+                update(`${lineHeader}${c.red(`❌ Failed to create directory: ${error.message}`)}`);
+                failed++;
+                return false;
+            });
+            if (!mkdirSuccess) {
+                return;
+            }
+            update(`${lineHeader}Writing file%dots%`);
+            const writeSuccess = await writeFile(fullPath, content)
+                .then(() => true)
+                .catch((error) => {
+                update(`${lineHeader}${c.red(`❌ Failed to write: ${error.message}`)}`);
+                failed++;
+                return false;
+            });
+            if (!writeSuccess) {
+                return;
+            }
+            downloaded++;
+            update(`${lineHeader}${c.green('✅ Downloaded')}`);
+        };
+    }, {
+        limit: 10,
+        verbose,
+        header: (progress, done, total, limit) => {
+            const percentage = Math.round(progress * 100);
+            if (progress === 1) {
+                return `${c.bold.green('📥 Download Complete')} ${done}/${total} (${percentage}%) - ${c.green(`✅ ${downloaded}`)} ${c.red(`❌ ${failed}`)}`;
+            }
+            return `${c.bold.cyan('📥 Downloading')} %earth% %dots% ${done}/${total} (${percentage}%) - Limit: ${limit}`;
+        },
+    });
+    logger('');
+    logger(c.bold.green(`✅ Downloaded: ${downloaded}, Failed: ${failed}`));
+}

package/dist/types.d.ts

@@ -0,0 +1,27 @@
+export interface ReplicateOptions {
+    urls: string[];
+    outputDir: string;
+    verbose?: boolean;
+    timeout?: number;
+    devices?: Record<string, {
+        width: number;
+        resolution?: number;
+    }>;
+    limit?: number;
+    only?: 'page' | 'resource';
+}
+export interface Resource {
+    url: string;
+    localPath: string;
+}
+export interface ChildProcessInput {
+    devices?: Record<string, {
+        width: number;
+        resolution?: number;
+    }>;
+    timeout?: number;
+}
+export interface ChildProcessResult {
+    url: string;
+    encodedUrls: string[];
+}

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@d-zero/replicator",
-	"version": "0.5.4",
+	"version": "0.6.1",
 	"description": "Replicate web pages with all their resources to local directories",
 	"author": "D-ZERO",
 	"license": "MIT",
@@ -25,9 +25,11 @@
 	},
 	"dependencies": {
 		"@d-zero/cli-core": "1.1.3",
-		"@d-zero/puppeteer-page-scan": "4.2.4",
-		"@d-zero/puppeteer-scroll": "3.0.10",
-		"@d-zero/shared": "0.11.0",
+		"@d-zero/dealer": "1.3.2",
+		"@d-zero/puppeteer-dealer": "0.5.9",
+		"@d-zero/puppeteer-page-scan": "4.2.5",
+		"@d-zero/puppeteer-scroll": "3.0.11",
+		"@d-zero/shared": "0.12.0",
 		"ansi-colors": "4.1.3",
 		"minimist": "1.2.8",
 		"puppeteer": "24.26.1"
@@ -35,5 +37,5 @@
 	"devDependencies": {
 		"@types/minimist": "1.2.5"
 	},
-	"gitHead": "8584d3f248d58be97ab008faf5ecd2bfbba32de0"
+	"gitHead": "bf9cfd1370947212367041214f68c8e8e6317b86"
 }