website-scrap-engine 0.8.6 → 0.9.0

package/README.md

@@ -1,42 +1,248 @@
 # website-scrap-engine
-Configurable website scraper in typescript.
+
+Configurable website scraper library in TypeScript. Consumers provide a `DownloadOptions` config (which includes a `ProcessingLifeCycle`) and instantiate a downloader to recursively scrape websites to local disk.
 
 ## Features
-- [x] Resource types
-- [x] Configurable process pipeline
-- [x] Options
-- [x] Logger
-- [x] Concurrent downloader
-- [x] Multi-thread processing (with native worker_thread)
-- [x] Process CSS
-- [x] Process HTML
-- [x] Process SiteMap (but not replace path in it)
-- [x] Configurable logging
-
-## Multi-thread processing
-Note: use multi-thread processing
-only if your process is cpu sensitive.
-
-* Main thread
-    * resource downloading in queue
-    * process after download
-    * save binary resources to disk
-    * send other resources to worker thread
-    * enqueue non-duplicated resource from worker thread
-* Worker thread
-    * receive downloaded resource from main thread
-    * process after download
-        * parse html, css, etc.
-    * collect referenced resources
-    * process and filter referenced resources before download
-    * send referenced resources to main thread
-    * save resources to disk
-
-## Pipeline life cycle
-* skip or redirect link
-* detect resource type
-* create
-* process before download
-* download
-* process after download
-* save to disk
+
+- Configurable processing pipeline with hook arrays at every stage
+- Single-thread and multi-thread (native `worker_threads`) downloaders
+- HTML, CSS, SVG, and sitemap parsing with automatic link discovery
+- CSS `url()` extraction and rewriting
+- `srcset`, Open Graph meta tags, inline styles, and SVG `xlink:href` support
+- Automatic URL-to-relative-path rewriting so saved sites work offline
+- Streaming download support for large binary resources
+- PQueue-based concurrency with runtime adjustment
+- URL deduplication with configurable search-param stripping
+- Configurable retry with exponential backoff, jitter, and `Retry-After` header support
+- Local `file://` source support for re-processing previously saved sites
+- Configurable logging via log4js with dedicated categories (`skip`, `retry`, `error`, `notFound`, etc.)
+
+## Installation
+
+```bash
+npm install website-scrap-engine
+```
+
+Requires Node.js >= 18.17.0.
+
+## Usage
+
+The downloader takes a path (or `file://` URL) to a module that default-exports a `DownloadOptions` object. This pattern allows worker threads to independently load the same configuration.
+
+**Step 1: Create an options module** (e.g. `my-options.js`)
+
+```ts
+import {lifeCycle, options, resource} from 'website-scrap-engine';
+
+const {defaultLifeCycle} = lifeCycle;
+const {defaultDownloadOptions} = options;
+const {ResourceType} = resource;
+
+const lc = defaultLifeCycle();
+
+// Example: skip binary resources deeper than depth 2
+lc.processBeforeDownload.push((res) => {
+  if (res.depth > 2 && res.type === ResourceType.Binary) return;
+  return res;
+});
+
+export default defaultDownloadOptions({
+  ...lc,
+  localRoot: '/path/to/save',
+  maxDepth: 3,
+  initialUrl: ['https://example.com'],
+});
+```
+
+**Step 2: Create and run the downloader**
+
+```ts
+import path from 'path';
+import {downloader} from 'website-scrap-engine';
+
+const {SingleThreadDownloader} = downloader;
+
+const d = new SingleThreadDownloader(
+  'file://' + path.resolve('my-options.js')
+);
+d.start();
+d.onIdle().then(() => d.dispose());
+```
+
+For CPU-intensive workloads, use `MultiThreadDownloader` instead (see [Multi-Thread Processing](#multi-thread-processing)).
+
+You can also pass override options as the second argument to the downloader constructor, which are merged into the options module's export:
+
+```ts
+new SingleThreadDownloader('file://' + path.resolve('my-options.js'), {
+  localRoot: '/different/path',
+  concurrency: 8,
+});
+```
+
+### Adapter Helpers
+
+The library provides adapter functions in `lifeCycle.adapter` for common customization patterns:
+
+| Adapter | Stage | Description |
+|---|---|---|
+| `skipProcess(fn)` | linkRedirect | Skip URLs matching a predicate |
+| `dropResource(fn)` | processBeforeDownload | Mark matching resources as discard-only (replace link but don't download) |
+| `preProcess(fn)` | processBeforeDownload | Inspect/modify resources before download |
+| `requestRedirect(fn)` | processBeforeDownload | Rewrite the download URL |
+| `redirectFilter(fn)` | processAfterDownload | Rewrite or discard redirect URLs |
+| `processHtml(fn)` | processAfterDownload | Transform the parsed HTML (cheerio `$`) |
+| `processHtmlAsync(fn)` | processAfterDownload | Async version of `processHtml` |
+
+```ts
+import {lifeCycle} from 'website-scrap-engine';
+
+const lc = lifeCycle.defaultLifeCycle();
+
+// Skip all URLs containing "/api/"
+lc.linkRedirect.push(lifeCycle.adapter.skipProcess(
+  (url) => url.includes('/api/')
+));
+
+// Drop images from download but still rewrite their links
+lc.processBeforeDownload.push(lifeCycle.adapter.dropResource(
+  (res) => res.type === ResourceType.Binary && res.url.endsWith('.png')
+));
+```
+
+## Architecture
+
+### Pipeline Life Cycle
+
+Resources are processed through a sequential pipeline of hook arrays. Each stage is an array of functions executed in order. Returning `void`/`undefined` from any function discards the resource from that stage onward.
+
+```
+init (once per downloader/worker startup)
+ |
+ v
+URL
+ |
+ v
+1. linkRedirect -----> skip or redirect URLs before processing
+ |
+ v
+2. detectResourceType -> determine type (Html, Css, Binary, Svg, SiteMap, etc.)
+ |
+ v
+3. createResource ----> build a Resource with save paths and relative replacement paths
+ |
+ v
+4. processBeforeDownload -> filter/modify resources; link replacement in parent happens after this
+ |
+ v
+5. download ----------> fetch resource via HTTP (loop ends early once body is set)
+ |
+ v
+6. processAfterDownload -> parse content, discover child resources via submit() callback
+ |
+ v
+7. saveToDisk --------> write to local filesystem
+ |
+ v
+dispose (once per downloader shutdown / worker exit)
+```
+
+Consumers extend the pipeline by prepending or appending functions to any stage array via `defaultLifeCycle()`. See [Usage](#usage) for examples.
+
+### Default Pipeline Handlers
+
+| Stage | Default handlers |
+|---|---|
+| linkRedirect | `skipLinks` - filters out non-HTTP URI schemes (mailto, javascript, data, etc.) |
+| detectResourceType | `detectResourceType` - infers type from element/context |
+| createResource | `createResource` - builds Resource with URL resolution, save path, and replace path |
+| download | `downloadResource`, `downloadStreamingResource`, `readOrCopyLocalResource` |
+| processAfterDownload | `processRedirectedUrl`, `processHtml`, `processHtmlMetaRefresh`, `processSvg`, `processCss`, `processSiteMap` |
+| saveToDisk | `saveHtmlToDisk`, `saveResourceToDisk` |
+
+### Resource Types
+
+Defined in `ResourceType` enum:
+
+| Type | Encoding | Description |
+|---|---|---|
+| `Binary` | null | Not parsed, saved as-is |
+| `Html` | utf8 | Parsed with cheerio, links discovered and rewritten |
+| `Css` | utf8 | CSS `url()` references extracted and rewritten |
+| `CssInline` | utf8 | Inline `<style>` blocks and `style` attributes |
+| `SiteMap` | utf8 | URLs discovered but not rewritten |
+| `Svg` | utf8 | Parsed with cheerio (same as HTML) |
+| `StreamingBinary` | null | Streamed directly to disk, for large files |
+
+### HTML Source Definitions
+
+The scraper discovers linked resources from HTML using configurable source definitions. The defaults cover:
+
+- Images: `img[src]`, `img[srcset]`, `picture source[srcset]`
+- Styles: `link[rel="stylesheet"]`, `<style>` blocks, `[style]` attributes
+- Scripts: `script[src]`
+- Links: `a[href]`, `frame[src]`, `iframe[src]`
+- Media: `video[src]`, `video[poster]`, `audio[src]`, `source[src]`, `track[src]`
+- SVG: `*[xlink:href]`, `*[href]`
+- Meta: `meta[property="og:image"]`, `og:audio`, `og:video` and their variants
+- Other: `embed[src]`, `object[data]`, `input[src]`, `[background]`, `link[rel*="icon"]`, `link[rel*="preload"]`
+
+Override via `options.sources` with an array of `{selector, attr, type}` definitions.
+
+### Key Abstractions
+
+- **`Resource`** (`src/resource.ts`) - Central data object carrying URL, save path, replacement path, body, and metadata. `RawResource` is the serializable subset used for cross-thread communication.
+- **`PipelineExecutor`** (interface in `src/life-cycle/pipeline-executor.ts`, impl in `src/downloader/pipeline-executor-impl.ts`) - Orchestrates life cycle execution. `createAndProcessResource()` runs stages 1-4 in one call.
+- **`AbstractDownloader`** (`src/downloader/main.ts`) - Base class with PQueue-based concurrency, URL deduplication, and the download loop.
+- **`SingleThreadDownloader`** (`src/downloader/single.ts`) - Runs all pipeline stages in the main thread.
+- **`MultiThreadDownloader`** (`src/downloader/multi.ts`) - Downloads in main thread, sends to worker pool for post-processing.
+
+## Multi-Thread Processing
+
+Use multi-thread processing when post-download work (HTML/CSS parsing, link discovery) is CPU-intensive.
+
+**Main thread:**
+- Runs the download queue with PQueue concurrency control
+- Executes stages 1-5 (linkRedirect through download)
+- Transfers downloaded resources to worker threads
+- Receives discovered child resources back and enqueues non-duplicates
+
+**Worker threads:**
+- Receive downloaded resources from the main thread
+- Execute stages 6-7 (processAfterDownload + saveToDisk)
+- Parse HTML/CSS/SVG, discover child resources
+- Run stages 1-4 on discovered children to prepare them
+- Send prepared child resources back to the main thread as `RawResource[]`
+
+Worker count defaults to `Math.min(concurrency, workerCount)`. The worker pool uses a 2-pass water-fill algorithm to balance tasks across workers by load.
+
+## Logging
+
+The library uses log4js with dedicated logger categories:
+
+| Logger | Purpose |
+|---|---|
+| `skip` | Resources filtered/discarded at any pipeline stage |
+| `skipExternal` | External resources skipped by scope |
+| `retry` | HTTP retry attempts with backoff details |
+| `error` | Download and processing errors |
+| `notFound` | 404 responses |
+| `request` / `response` | HTTP request/response logging |
+| `complete` | Successfully processed resources |
+| `mkdir` | Directory creation |
+| `adjustConcurrency` | Runtime concurrency changes |
+
+Configure logging via `options.configureLogger` and `options.logSubDir`.
+
+## Key Dependencies
+
+- **cheerio** - HTML/SVG parsing and manipulation
+- **got** - HTTP client with retry logic
+- **p-queue** - Download concurrency control
+- **urijs** - URL resolution and path generation
+- **css-url-parser** - CSS `url()` extraction
+- **srcset** - `srcset` attribute parsing
+
+## License
+
+ISC

package/lib/downloader/main.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"main.d.ts","sourceRoot":"","sources":["../../src/downloader/main.ts"],"names":[],"mappings":"AAAA,OAAO,MAAM,MAAM,SAAS,CAAC;AAG7B,OAAO,KAAK,EAAC,eAAe,EAAE,qBAAqB,EAAC,MAAM,eAAe,CAAC;AAE1E,OAAO,KAAK,EAAC,WAAW,EAAW,MAAM,gBAAgB,CAAC;AAI1D,OAAO,KAAK,EAAC,eAAe,EAAE,kBAAkB,EAAC,MAAM,YAAY,CAAC;AACpE,OAAO,EAAC,oBAAoB,EAAC,MAAM,6BAA6B,CAAC;AAEjE,8BAAsB,kBAAmB,YAAW,kBAAkB;IAkBvC,aAAa,EAAE,MAAM;IAjBlD,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;IACvB,QAAQ,CAAC,aAAa,EAAE,OAAO,CAAC,eAAe,CAAC,CAAC;IACjD,QAAQ,CAAC,gBAAgB,CAAC,EAAE,OAAO,CAAC,qBAAqB,CAAC,GAAG;QAAE,YAAY,CAAC,EAAE,MAAM,CAAA;KAAE,CAAC;IACvF,QAAQ,CAAC,EAAE,eAAe,CAAC;IAC3B,OAAO,EAAE,OAAO,CAAC;IACjB,SAAS,CAAC,EAAE,oBAAoB,CAAC;IACjC,YAAY,EAAE,OAAO,CAAC,IAAI,CAAC,CAAC;IAC5B,QAAQ,CAAC,aAAa,EAAE,GAAG,CAAC,MAAM,CAAC,CAAqB;IACxD,QAAQ,CAAC,SAAS,EAAE,GAAG,CAAC,MAAM,CAAC,CAAqB;IACpD,QAAQ,CAAC,IAAI,EAAE,eAAe,CAK5B;IACF,WAAW,EAAE,UAAU,CAAC,OAAO,WAAW,CAAC,GAAG,IAAI,CAAa;IAE/D,SAAS,aAAoB,aAAa,EAAE,MAAM,EAChD,eAAe,CAAC,EAAE,OAAO,CAAC,qBAAqB,CAAC,GAAG;QAAE,YAAY,CAAC,EAAE,MAAM,CAAA;KAAE;IAoB9E,SAAS,CAAC,aAAa,CAAC,OAAO,EAAE,eAAe,GAAG,OAAO,CAAC,IAAI,CAAC;IAIhE,IAAI,OAAO,IAAI,eAAe,CAK7B;IAED,IAAI,QAAQ,IAAI,oBAAoB,CAKnC;IAED,IAAI,WAAW,IAAI,MAAM,CAExB;IAED,IAAI,WAAW,CAAC,cAAc,EAAE,MAAM,EAErC;IAED,IAAI,SAAS,IAAI,MAAM,CAEtB;IAED,IAAI,YAAY,IAAI,MAAM,CAEzB;IAEK,kBAAkB,CAAC,MAAM,EAAE,MAAM,EAAE,GAAG,OAAO,CAAC,IAAI,CAAC;IA2BzD,SAAS,CAAC,qBAAqB,CAAC,GAAG,EAAE,WAAW,GAAG,OAAO,GAAG,IAAI;IAwBjE,QAAQ,CAAC,0BAA0B,CAAC,GAAG,EAAE,WAAW,GAAG,OAAO,CAAC,OAAO,GAAG,IAAI,CAAC;IAE9E,oBAAoB,CAAC,GAAG,EAAE,WAAW,GAAG,OAAO,GAAG,IAAI;IAStD,WAAW,CAAC,GAAG,EAAE,KAAK,GAAG,OAAO,GAAG,IAAI,EAAE,KAAK,EAAE,MAAM,EAAE,QAAQ,EAAE,WAAW,GAAG,IAAI;IAapF,IAAI,eAAe,IAAI,MAAM,CAE5B;IAED,KAAK,IAAI,IAAI;IAcb,IAAI,IAAI,IAAI;IAOZ,MAAM,IAAI,OAAO,CAAC,IAAI,CAAC;IAIjB,OAAO,IAAI,OAAO,CAAC,IAAI,CAAC;CAM/B"}
+{"version":3,"file":"main.d.ts","sourceRoot":"","sources":["../../src/downloader/main.ts"],"names":[],"mappings":"AAAA,OAAO,MAAM,MAAM,SAAS,CAAC;AAE7B,OAAO,KAAK,EAAC,eAAe,EAAE,qBAAqB,EAAC,MAAM,eAAe,CAAC;AAE1E,OAAO,KAAK,EAAC,WAAW,EAAW,MAAM,gBAAgB,CAAC;AAM1D,OAAO,KAAK,EAAC,eAAe,EAAE,kBAAkB,EAAC,MAAM,YAAY,CAAC;AACpE,OAAO,EAAC,oBAAoB,EAAC,MAAM,6BAA6B,CAAC;AAGjE,8BAAsB,kBAAmB,YAAW,kBAAkB;IAkBvC,aAAa,EAAE,MAAM;IAjBlD,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;IACvB,QAAQ,CAAC,aAAa,EAAE,OAAO,CAAC,eAAe,CAAC,CAAC;IACjD,QAAQ,CAAC,gBAAgB,CAAC,EAAE,OAAO,CAAC,qBAAqB,CAAC,GAAG;QAAE,YAAY,CAAC,EAAE,MAAM,CAAA;KAAE,CAAC;IACvF,QAAQ,CAAC,EAAE,eAAe,CAAC;IAC3B,OAAO,EAAE,OAAO,CAAC;IACjB,SAAS,CAAC,EAAE,oBAAoB,CAAC;IACjC,YAAY,EAAE,OAAO,CAAC,IAAI,CAAC,CAAC;IAC5B,QAAQ,CAAC,aAAa,EAAE,GAAG,CAAC,MAAM,CAAC,CAAqB;IACxD,QAAQ,CAAC,SAAS,EAAE,GAAG,CAAC,MAAM,CAAC,CAAqB;IACpD,QAAQ,CAAC,IAAI,EAAE,eAAe,CAK5B;IACF,WAAW,EAAE,UAAU,CAAC,OAAO,WAAW,CAAC,GAAG,IAAI,CAAa;IAE/D,SAAS,aAAoB,aAAa,EAAE,MAAM,EAChD,eAAe,CAAC,EAAE,OAAO,CAAC,qBAAqB,CAAC,GAAG;QAAE,YAAY,CAAC,EAAE,MAAM,CAAA;KAAE;IAoB9E,SAAS,CAAC,aAAa,CAAC,OAAO,EAAE,eAAe,GAAG,OAAO,CAAC,IAAI,CAAC;IAIhE,IAAI,OAAO,IAAI,eAAe,CAK7B;IAED,IAAI,QAAQ,IAAI,oBAAoB,CAKnC;IAED,IAAI,WAAW,IAAI,MAAM,CAExB;IAED,IAAI,WAAW,CAAC,cAAc,EAAE,MAAM,EAErC;IAED,IAAI,SAAS,IAAI,MAAM,CAEtB;IAED,IAAI,YAAY,IAAI,MAAM,CAEzB;IAEK,kBAAkB,CAAC,MAAM,EAAE,MAAM,EAAE,GAAG,OAAO,CAAC,IAAI,CAAC;IA8BzD,SAAS,CAAC,qBAAqB,CAAC,GAAG,EAAE,WAAW,GAAG,OAAO,GAAG,IAAI;IAyBjE,QAAQ,CAAC,0BAA0B,CAAC,GAAG,EAAE,WAAW,GAAG,OAAO,CAAC,OAAO,GAAG,IAAI,CAAC;IAE9E,oBAAoB,CAAC,GAAG,EAAE,WAAW,GAAG,OAAO,GAAG,IAAI;IAStD,WAAW,CAAC,GAAG,EAAE,KAAK,GAAG,OAAO,GAAG,IAAI,EAAE,KAAK,EAAE,MAAM,EAAE,QAAQ,EAAE,WAAW,GAAG,IAAI;IAQpF,IAAI,eAAe,IAAI,MAAM,CAE5B;IAED,KAAK,IAAI,IAAI;IAcb,IAAI,IAAI,IAAI;IAOZ,MAAM,IAAI,OAAO,CAAC,IAAI,CAAC;IAIjB,OAAO,IAAI,OAAO,CAAC,IAAI,CAAC;CAM/B"}

package/lib/downloader/main.js

@@ -2,7 +2,9 @@ import PQueue from 'p-queue';
 import URI from 'urijs';
 import { mergeOverrideOptions } from '../options.js';
 import { normalizeResource, ResourceType } from '../resource.js';
-import { error, notFound, skip } from '../logger/logger.js';
+import { skip } from '../logger/logger.js';
+import { setLogger } from '../logger/logger.js';
+import { createDefaultLogger } from '../logger/default-logger.js';
 import { importDefaultFromPath } from '../util.js';
 import { PipelineExecutorImpl } from './pipeline-executor-impl.js';
 export class AbstractDownloader {
@@ -23,12 +25,13 @@ export class AbstractDownloader {
         this.queue = new PQueue({ concurrency: 2 });
         this._isInit = false;
         this._initOptions = this._asyncOptions.then(options => {
+            var _a;
             options = mergeOverrideOptions(options, this._overrideOptions);
             this._options = options;
             // https://github.com/website-local/website-scrap-engine/issues/1113
             this.queue.concurrency = options.concurrency;
             this._pipeline = new PipelineExecutorImpl(options, options.req, options);
-            options.configureLogger(options.localRoot, options.logSubDir || '');
+            setLogger(((_a = options.createLogger) !== null && _a !== void 0 ? _a : createDefaultLogger)(options));
             return this._internalInit(options).then(() => {
                 this._isInit = true;
             });
@@ -68,9 +71,12 @@ export class AbstractDownloader {
             await this._initOptions;
         }
         const pipeline = this.pipeline;
-        await pipeline.init(pipeline, this);
+        const submit = (url) => {
+            urlArr.push(url);
+        };
+        await pipeline.init(pipeline, this, submit);
         // noinspection DuplicatedCode
-        for (let i = 0, l = urlArr.length; i < l; i++) {
+        for (let i = 0; i < urlArr.length; i++) {
             let url = urlArr[i];
             url = await pipeline.linkRedirect(url, null, null);
             if (!url)
@@ -94,6 +100,7 @@ export class AbstractDownloader {
         // noinspection DuplicatedCode
         if (res.depth > this.options.maxDepth) {
             skip.info('skipped max depth', res.url, res.refUrl, res.depth);
+            this.pipeline.notifyStatusChange(res, 'dispose');
             return false;
         }
         let url;
@@ -123,18 +130,10 @@ export class AbstractDownloader {
         }
     }
     handleError(err, cause, resource) {
-        var _a;
-        // force cast in case of typescript 4.4
-        if (err && err.name === 'HTTPError' &&
-            ((_a = err === null || err === void 0 ? void 0 : err.response) === null || _a === void 0 ? void 0 : _a.statusCode) === 404) {
-            notFound.error(resource.url, resource.downloadLink, resource.refUrl);
-        }
-        else if (err) {
-            error.error(cause, resource.url, resource.downloadLink, resource.refUrl, err);
-        }
-        else {
-            error.error(cause, resource.url, resource.downloadLink, resource.refUrl);
-        }
+        resource.meta = resource.meta || {};
+        resource.meta['error'] = err;
+        resource.meta['errorCause'] = cause;
+        this.pipeline.notifyStatusChange(resource, 'error');
     }
     get downloadedCount() {
         return this.downloadedUrl.size;

package/lib/downloader/main.js.map

@@ -1 +1 @@
-{"version":3,"file":"main.js","sourceRoot":"","sources":["../../src/downloader/main.ts"],"names":[],"mappings":"AAAA,OAAO,MAAM,MAAM,SAAS,CAAC;AAE7B,OAAO,GAAG,MAAM,OAAO,CAAC;AAExB,OAAO,EAAC,oBAAoB,EAAC,MAAM,eAAe,CAAC;AAEnD,OAAO,EAAC,iBAAiB,EAAE,YAAY,EAAC,MAAM,gBAAgB,CAAC;AAC/D,OAAO,EAAC,KAAK,EAAE,QAAQ,EAAE,IAAI,EAAC,MAAM,qBAAqB,CAAC;AAC1D,OAAO,EAAC,qBAAqB,EAAC,MAAM,YAAY,CAAC;AAEjD,OAAO,EAAC,oBAAoB,EAAC,MAAM,6BAA6B,CAAC;AAEjE,MAAM,OAAgB,kBAAkB;IAkBtC,YAA6B,aAAqB,EAChD,eAA4E;QADjD,kBAAa,GAAb,aAAa,CAAQ;QAVzC,kBAAa,GAAgB,IAAI,GAAG,EAAU,CAAC;QAC/C,cAAS,GAAgB,IAAI,GAAG,EAAU,CAAC;QAC3C,SAAI,GAAoB;YAC/B,kBAAkB,EAAE,CAAC;YACrB,gBAAgB,EAAE,CAAC;YACnB,eAAe,EAAE,CAAC;YAClB,oBAAoB,EAAE,CAAC;SACxB,CAAC;QACF,gBAAW,GAA0C,SAAS,CAAC;QAI7D,IAAI,CAAC,aAAa,GAAG,qBAAqB,CAAC,aAAa,CAAC,CAAC;QAC1D,IAAI,CAAC,gBAAgB,GAAG,eAAe,CAAC;QACxC,6CAA6C;QAC7C,IAAI,CAAC,KAAK,GAAG,IAAI,MAAM,CAAC,EAAC,WAAW,EAAE,CAAC,EAAC,CAAC,CAAC;QAC1C,IAAI,CAAC,OAAO,GAAG,KAAK,CAAC;QACrB,IAAI,CAAC,YAAY,GAAG,IAAI,CAAC,aAAa,CAAC,IAAI,CAAC,OAAO,CAAC,EAAE;YACpD,OAAO,GAAG,oBAAoB,CAAC,OAAO,EAAE,IAAI,CAAC,gBAAgB,CAAC,CAAC;YAC/D,IAAI,CAAC,QAAQ,GAAG,OAAO,CAAC;YACxB,oEAAoE;YACpE,IAAI,CAAC,KAAK,CAAC,WAAW,GAAG,OAAO,CAAC,WAAW,CAAC;YAC7C,IAAI,CAAC,SAAS,GAAG,IAAI,oBAAoB,CAAC,OAAO,EAAE,OAAO,CAAC,GAAG,EAAE,OAAO,CAAC,CAAC;YACzE,OAAO,CAAC,eAAe,CAAC,OAAO,CAAC,SAAS,EAAE,OAAO,CAAC,SAAS,IAAI,EAAE,CAAC,CAAC;YACpE,OAAO,IAAI,CAAC,aAAa,CAAC,OAAO,CAAC,CAAC,IAAI,CAAC,GAAG,EAAE;gBAC3C,IAAI,CAAC,OAAO,GAAG,IAAI,CAAC;YACtB,CAAC,CAAC,CAAC;QACL,CAAC,CAAC,CAAC;IACL,CAAC;IAED,6DAA6D;IACnD,aAAa,CAAC,OAAwB;QAC9C,OAAO,OAAO,CAAC,OAAO,EAAE,CAAC;IAC3B,CAAC;IAED,IAAI,OAAO;QACT,IAAI,IAAI,CAAC,QAAQ,EAAE,CAAC;YAClB,OAAO,IAAI,CAAC,QAAQ,CAAC;QACvB,CAAC;QACD,MAAM,IAAI,SAAS,CAAC,qCAAqC,CAAC,CAAC;IAC7D,CAAC;IAED,IAAI,QAAQ;QACV,IAAI,IAAI,CAAC,SAAS,EAAE,CAAC;YACnB,OAAO,IAAI,CAAC,SAAS,CAAC;QACxB,CAAC;QACD,MAAM,IAAI,SAAS,CAAC,qCAAqC,CAAC,CAAC;IAC7D,CAAC;IAED,IAAI,WAAW;QACb,OAAO,IAAI,CAAC,KAAK,CAAC,WAAW,CAAC;IAChC,CAAC;IAED,IAAI,WAAW,CAAC,cAAsB;QACpC,IAAI,CAAC,KAAK,CAAC,WAAW,GAAG,cAAc,CAAC;IAC1C,CAAC;IAED,IAAI,SAAS;QACX,OAAO,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC;IACzB,CAAC;IAED,IAAI,YAAY;QACd,OAAO,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC;IAC5B,CAAC;IAED,KAAK,CAAC,kBAAkB,CAAC,MAAgB;QACvC,IAAI,CAAC,IAAI,CAAC,SAAS,EAAE,CAAC;YACpB,8CAA8C;YAC9C,MAAM,IAAI,CAAC,YAAY,CAAC;QAC1B,CAAC;QACD,MAAM,QAAQ,GAAG,IAAI,CAAC,QAAQ,CAAC;QAC/B,MAAM,QAAQ,CAAC,IAAI,CAAC,QAAQ,EAAE,IAAI,CAAC,CAAC;QACpC,8BAA8B;QAC9B,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,MAAM,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC;YAC9C,IAAI,GAAG,GAAkB,MAAM,CAAC,CAAC,CAAC,CAAC;YACnC,GAAG,GAAG,MAAM,QAAQ,CAAC,YAAY,CAAC,GAAG,EAAE,IAAI,EAAE,IAAI,CAAC,CAAC;YACnD,IAAI,CAAC,GAAG;gBAAE,SAAS;YACnB,MAAM,IAAI,GAAwB,MAAM,QAAQ,CAAC,kBAAkB,CACjE,GAAG,EAAE,YAAY,CAAC,IAAI,EAAE,IAAI,EAAE,IAAI,CAAC,CAAC;YACtC,IAAI,CAAC,IAAI;gBAAE,SAAS;YACpB,IAAI,CAAC,GAAoB,MAAM,QAAQ,CAAC,cAAc,CACpD,IAAI,EAAE,CAAC,EAAE,GAAG,EAAE,GAAG,EACjB,SAAS,EAAE,SAAS,EAAE,SAAS,EAAE,IAAI,CAAC,CAAC;YACzC,IAAI,CAAC,CAAC;gBAAE,SAAS;YACjB,CAAC,GAAG,MAAM,QAAQ,CAAC,qBAAqB,CAAC,CAAC,EAAE,IAAI,EAAE,IAAI,CAAC,CAAC;YACxD,IAAI,CAAC,CAAC;gBAAE,SAAS;YACjB,IAAI,CAAC,CAAC,CAAC,6BAA6B,EAAE,CAAC;gBACrC,IAAI,CAAC,oBAAoB,CAAC,CAAC,CAAC,CAAC;YAC/B,CAAC;QACH,CAAC;IACH,CAAC;IAES,qBAAqB,CAAC,GAAgB;;QAC9C,8BAA8B;QAC9B,IAAI,GAAG,CAAC,KAAK,GAAG,IAAI,CAAC,OAAO,CAAC,QAAQ,EAAE,CAAC;YACtC,IAAI,CAAC,IAAI,CAAC,mBAAmB,EAAE,GAAG,CAAC,GAAG,EAAE,GAAG,CAAC,MAAM,EAAE,GAAG,CAAC,KAAK,CAAC,CAAC;YAC/D,OAAO,KAAK,CAAC;QACf,CAAC;QACD,IAAI,GAAW,CAAC;QAChB,MAAM,GAAG,GAAQ,CAAC,CAAA,MAAC,GAAgB,aAAhB,GAAG,uBAAH,GAAG,CAAe,GAAG,0CAAE,KAAK,EAAE,KAAI,GAAG,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;QAC5E,IAAI,IAAI,CAAC,OAAO,CAAC,sBAAsB,EAAE,CAAC;YACxC,GAAG,GAAG,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,QAAQ,EAAE,CAAC;QAClC,CAAC;aAAM,CAAC;YACN,GAAG,GAAG,GAAG,CAAC,QAAQ,EAAE,CAAC;QACvB,CAAC;QACD,IAAI,IAAI,CAAC,SAAS,CAAC,GAAG,CAAC,GAAG,CAAC,EAAE,CAAC;YAC5B,OAAO,KAAK,CAAC;QACf,CAAC;QACD,IAAI,CAAC,SAAS,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;QACxB,MAAM,QAAQ,GAAa,iBAAiB,CAAC,GAAG,CAAC,CAAC;QAClD,qBAAqB;QACrB,wCAAwC;QACxC,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,GAAG,EAAE,CAAC,IAAI,OAAO,CAAC,CAAC,CAAC,EAAE,CAAC,YAAY,CAChD,GAAG,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,0BAA0B,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IAC1D,CAAC;IAID,oBAAoB,CAAC,GAAgB;QACnC,IAAI,CAAC;YACH,OAAO,IAAI,CAAC,qBAAqB,CAAC,GAAG,CAAC,CAAC;QACzC,CAAC;QAAC,OAAO,CAAC,EAAE,CAAC;YACX,IAAI,CAAC,WAAW,CAAC,CAAC,EAAE,iBAAiB,EAAE,GAAG,CAAC,CAAC;YAC5C,OAAO,KAAK,CAAC;QACf,CAAC;IACH,CAAC;IAED,WAAW,CAAC,GAA2B,EAAE,KAAa,EAAE,QAAqB;;QAC3E,uCAAuC;QACvC,IAAI,GAAG,IAAK,GAAuB,CAAC,IAAI,KAAK,WAAW;YACtD,CAAA,MAAC,GAAiB,aAAjB,GAAG,uBAAH,GAAG,CAAgB,QAAQ,0CAAE,UAAU,MAAK,GAAG,EAAE,CAAC;YACnD,QAAQ,CAAC,KAAK,CAAC,QAAQ,CAAC,GAAG,EAAE,QAAQ,CAAC,YAAY,EAAE,QAAQ,CAAC,MAAM,CAAC,CAAC;QACvE,CAAC;aAAM,IAAI,GAAG,EAAE,CAAC;YACf,KAAK,CAAC,KAAK,CAAC,KAAK,EAAE,QAAQ,CAAC,GAAG,EAAE,QAAQ,CAAC,YAAY,EAAE,QAAQ,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC;QAChF,CAAC;aAAM,CAAC;YACN,KAAK,CAAC,KAAK,CAAC,KAAK,EAAE,QAAQ,CAAC,GAAG,EAAE,QAAQ,CAAC,YAAY,EAAE,QAAQ,CAAC,MAAM,CAAC,CAAC;QAC3E,CAAC;IACH,CAAC;IAGD,IAAI,eAAe;QACjB,OAAO,IAAI,CAAC,aAAa,CAAC,IAAI,CAAC;IACjC,CAAC;IAED,KAAK;QACH,IAAI,CAAC,YAAY,CAAC,IAAI,CAAC,GAAG,EAAE;YAC1B,IAAI,OAAO,IAAI,CAAC,OAAO,CAAC,qBAAqB,KAAK,UAAU,EAAE,CAAC;gBAC7D,IAAI,IAAI,CAAC,WAAW,EAAE,CAAC;oBACrB,aAAa,CAAC,IAAI,CAAC,WAAW,CAAC,CAAC;gBAClC,CAAC;gBACD,IAAI,CAAC,WAAW,GAAG,WAAW,CAC5B,GAAG,EAAE,eAAC,OAAA,MAAA,MAAA,IAAI,CAAC,OAAO,EAAC,qBAAqB,mDAAG,IAAI,CAAC,CAAA,EAAA,EAChD,IAAI,CAAC,OAAO,CAAC,uBAAuB,IAAI,KAAK,CAAC,CAAC;YACnD,CAAC;YACD,IAAI,CAAC,KAAK,CAAC,KAAK,EAAE,CAAC;QACrB,CAAC,CAAC,CAAC;IACL,CAAC;IAED,IAAI;QACF,IAAI,IAAI,CAAC,WAAW,EAAE,CAAC;YACrB,aAAa,CAAC,IAAI,CAAC,WAAW,CAAC,CAAC;QAClC,CAAC;QACD,IAAI,CAAC,KAAK,CAAC,KAAK,EAAE,CAAC;IACrB,CAAC;IAED,MAAM;QACJ,OAAO,IAAI,CAAC,YAAY,CAAC,IAAI,CAAC,GAAG,EAAE,CAAC,IAAI,CAAC,KAAK,CAAC,MAAM,EAAE,CAAC,CAAC;IAC3D,CAAC;IAED,KAAK,CAAC,OAAO;;QACX,IAAI,CAAC,IAAI,EAAE,CAAC;QACZ,IAAI,CAAC,KAAK,CAAC,KAAK,EAAE,CAAC;QACnB,MAAM,CAAA,MAAA,IAAI,CAAC,QAAQ,0CAAE,OAAO,CAAC,IAAI,CAAC,QAAQ,EAAE,IAAI,CAAC,CAAA,CAAC;IACpD,CAAC;CAEF"}
+{"version":3,"file":"main.js","sourceRoot":"","sources":["../../src/downloader/main.ts"],"names":[],"mappings":"AAAA,OAAO,MAAM,MAAM,SAAS,CAAC;AAC7B,OAAO,GAAG,MAAM,OAAO,CAAC;AAExB,OAAO,EAAC,oBAAoB,EAAC,MAAM,eAAe,CAAC;AAEnD,OAAO,EAAC,iBAAiB,EAAE,YAAY,EAAC,MAAM,gBAAgB,CAAC;AAC/D,OAAO,EAAC,IAAI,EAAC,MAAM,qBAAqB,CAAC;AACzC,OAAO,EAAC,SAAS,EAAC,MAAM,qBAAqB,CAAC;AAC9C,OAAO,EAAC,mBAAmB,EAAC,MAAM,6BAA6B,CAAC;AAChE,OAAO,EAAC,qBAAqB,EAAC,MAAM,YAAY,CAAC;AAEjD,OAAO,EAAC,oBAAoB,EAAC,MAAM,6BAA6B,CAAC;AAGjE,MAAM,OAAgB,kBAAkB;IAkBtC,YAA6B,aAAqB,EAChD,eAA4E;QADjD,kBAAa,GAAb,aAAa,CAAQ;QAVzC,kBAAa,GAAgB,IAAI,GAAG,EAAU,CAAC;QAC/C,cAAS,GAAgB,IAAI,GAAG,EAAU,CAAC;QAC3C,SAAI,GAAoB;YAC/B,kBAAkB,EAAE,CAAC;YACrB,gBAAgB,EAAE,CAAC;YACnB,eAAe,EAAE,CAAC;YAClB,oBAAoB,EAAE,CAAC;SACxB,CAAC;QACF,gBAAW,GAA0C,SAAS,CAAC;QAI7D,IAAI,CAAC,aAAa,GAAG,qBAAqB,CAAC,aAAa,CAAC,CAAC;QAC1D,IAAI,CAAC,gBAAgB,GAAG,eAAe,CAAC;QACxC,6CAA6C;QAC7C,IAAI,CAAC,KAAK,GAAG,IAAI,MAAM,CAAC,EAAC,WAAW,EAAE,CAAC,EAAC,CAAC,CAAC;QAC1C,IAAI,CAAC,OAAO,GAAG,KAAK,CAAC;QACrB,IAAI,CAAC,YAAY,GAAG,IAAI,CAAC,aAAa,CAAC,IAAI,CAAC,OAAO,CAAC,EAAE;;YACpD,OAAO,GAAG,oBAAoB,CAAC,OAAO,EAAE,IAAI,CAAC,gBAAgB,CAAC,CAAC;YAC/D,IAAI,CAAC,QAAQ,GAAG,OAAO,CAAC;YACxB,oEAAoE;YACpE,IAAI,CAAC,KAAK,CAAC,WAAW,GAAG,OAAO,CAAC,WAAW,CAAC;YAC7C,IAAI,CAAC,SAAS,GAAG,IAAI,oBAAoB,CAAC,OAAO,EAAE,OAAO,CAAC,GAAG,EAAE,OAAO,CAAC,CAAC;YACzE,SAAS,CAAC,CAAC,MAAA,OAAO,CAAC,YAAY,mCAAI,mBAAmB,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC;YAClE,OAAO,IAAI,CAAC,aAAa,CAAC,OAAO,CAAC,CAAC,IAAI,CAAC,GAAG,EAAE;gBAC3C,IAAI,CAAC,OAAO,GAAG,IAAI,CAAC;YACtB,CAAC,CAAC,CAAC;QACL,CAAC,CAAC,CAAC;IACL,CAAC;IAED,6DAA6D;IACnD,aAAa,CAAC,OAAwB;QAC9C,OAAO,OAAO,CAAC,OAAO,EAAE,CAAC;IAC3B,CAAC;IAED,IAAI,OAAO;QACT,IAAI,IAAI,CAAC,QAAQ,EAAE,CAAC;YAClB,OAAO,IAAI,CAAC,QAAQ,CAAC;QACvB,CAAC;QACD,MAAM,IAAI,SAAS,CAAC,qCAAqC,CAAC,CAAC;IAC7D,CAAC;IAED,IAAI,QAAQ;QACV,IAAI,IAAI,CAAC,SAAS,EAAE,CAAC;YACnB,OAAO,IAAI,CAAC,SAAS,CAAC;QACxB,CAAC;QACD,MAAM,IAAI,SAAS,CAAC,qCAAqC,CAAC,CAAC;IAC7D,CAAC;IAED,IAAI,WAAW;QACb,OAAO,IAAI,CAAC,KAAK,CAAC,WAAW,CAAC;IAChC,CAAC;IAED,IAAI,WAAW,CAAC,cAAsB;QACpC,IAAI,CAAC,KAAK,CAAC,WAAW,GAAG,cAAc,CAAC;IAC1C,CAAC;IAED,IAAI,SAAS;QACX,OAAO,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC;IACzB,CAAC;IAED,IAAI,YAAY;QACd,OAAO,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC;IAC5B,CAAC;IAED,KAAK,CAAC,kBAAkB,CAAC,MAAgB;QACvC,IAAI,CAAC,IAAI,CAAC,SAAS,EAAE,CAAC;YACpB,8CAA8C;YAC9C,MAAM,IAAI,CAAC,YAAY,CAAC;QAC1B,CAAC;QACD,MAAM,QAAQ,GAAG,IAAI,CAAC,QAAQ,CAAC;QAC/B,MAAM,MAAM,GAAmB,CAAC,GAAW,EAAE,EAAE;YAC7C,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;QACnB,CAAC,CAAC;QACF,MAAM,QAAQ,CAAC,IAAI,CAAC,QAAQ,EAAE,IAAI,EAAE,MAAM,CAAC,CAAC;QAC5C,8BAA8B;QAC9B,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,MAAM,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;YACvC,IAAI,GAAG,GAAkB,MAAM,CAAC,CAAC,CAAC,CAAC;YACnC,GAAG,GAAG,MAAM,QAAQ,CAAC,YAAY,CAAC,GAAG,EAAE,IAAI,EAAE,IAAI,CAAC,CAAC;YACnD,IAAI,CAAC,GAAG;gBAAE,SAAS;YACnB,MAAM,IAAI,GAAwB,MAAM,QAAQ,CAAC,kBAAkB,CACjE,GAAG,EAAE,YAAY,CAAC,IAAI,EAAE,IAAI,EAAE,IAAI,CAAC,CAAC;YACtC,IAAI,CAAC,IAAI;gBAAE,SAAS;YACpB,IAAI,CAAC,GAAoB,MAAM,QAAQ,CAAC,cAAc,CACpD,IAAI,EAAE,CAAC,EAAE,GAAG,EAAE,GAAG,EACjB,SAAS,EAAE,SAAS,EAAE,SAAS,EAAE,IAAI,CAAC,CAAC;YACzC,IAAI,CAAC,CAAC;gBAAE,SAAS;YACjB,CAAC,GAAG,MAAM,QAAQ,CAAC,qBAAqB,CAAC,CAAC,EAAE,IAAI,EAAE,IAAI,CAAC,CAAC;YACxD,IAAI,CAAC,CAAC;gBAAE,SAAS;YACjB,IAAI,CAAC,CAAC,CAAC,6BAA6B,EAAE,CAAC;gBACrC,IAAI,CAAC,oBAAoB,CAAC,CAAC,CAAC,CAAC;YAC/B,CAAC;QACH,CAAC;IACH,CAAC;IAES,qBAAqB,CAAC,GAAgB;;QAC9C,8BAA8B;QAC9B,IAAI,GAAG,CAAC,KAAK,GAAG,IAAI,CAAC,OAAO,CAAC,QAAQ,EAAE,CAAC;YACtC,IAAI,CAAC,IAAI,CAAC,mBAAmB,EAAE,GAAG,CAAC,GAAG,EAAE,GAAG,CAAC,MAAM,EAAE,GAAG,CAAC,KAAK,CAAC,CAAC;YAC/D,IAAI,CAAC,QAAQ,CAAC,kBAAkB,CAAC,GAAG,EAAE,SAAS,CAAC,CAAC;YACjD,OAAO,KAAK,CAAC;QACf,CAAC;QACD,IAAI,GAAW,CAAC;QAChB,MAAM,GAAG,GAAQ,CAAC,CAAA,MAAC,GAAgB,aAAhB,GAAG,uBAAH,GAAG,CAAe,GAAG,0CAAE,KAAK,EAAE,KAAI,GAAG,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;QAC5E,IAAI,IAAI,CAAC,OAAO,CAAC,sBAAsB,EAAE,CAAC;YACxC,GAAG,GAAG,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,QAAQ,EAAE,CAAC;QAClC,CAAC;aAAM,CAAC;YACN,GAAG,GAAG,GAAG,CAAC,QAAQ,EAAE,CAAC;QACvB,CAAC;QACD,IAAI,IAAI,CAAC,SAAS,CAAC,GAAG,CAAC,GAAG,CAAC,EAAE,CAAC;YAC5B,OAAO,KAAK,CAAC;QACf,CAAC;QACD,IAAI,CAAC,SAAS,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;QACxB,MAAM,QAAQ,GAAa,iBAAiB,CAAC,GAAG,CAAC,CAAC;QAClD,qBAAqB;QACrB,wCAAwC;QACxC,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,GAAG,EAAE,CAAC,IAAI,OAAO,CAAC,CAAC,CAAC,EAAE,CAAC,YAAY,CAChD,GAAG,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,0BAA0B,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IAC1D,CAAC;IAID,oBAAoB,CAAC,GAAgB;QACnC,IAAI,CAAC;YACH,OAAO,IAAI,CAAC,qBAAqB,CAAC,GAAG,CAAC,CAAC;QACzC,CAAC;QAAC,OAAO,CAAC,EAAE,CAAC;YACX,IAAI,CAAC,WAAW,CAAC,CAAC,EAAE,iBAAiB,EAAE,GAAG,CAAC,CAAC;YAC5C,OAAO,KAAK,CAAC;QACf,CAAC;IACH,CAAC;IAED,WAAW,CAAC,GAA2B,EAAE,KAAa,EAAE,QAAqB;QAC3E,QAAQ,CAAC,IAAI,GAAG,QAAQ,CAAC,IAAI,IAAI,EAAE,CAAC;QACpC,QAAQ,CAAC,IAAI,CAAC,OAAO,CAAC,GAAG,GAAG,CAAC;QAC7B,QAAQ,CAAC,IAAI,CAAC,YAAY,CAAC,GAAG,KAAK,CAAC;QACpC,IAAI,CAAC,QAAQ,CAAC,kBAAkB,CAAC,QAAQ,EAAE,OAAO,CAAC,CAAC;IACtD,CAAC;IAGD,IAAI,eAAe;QACjB,OAAO,IAAI,CAAC,aAAa,CAAC,IAAI,CAAC;IACjC,CAAC;IAED,KAAK;QACH,IAAI,CAAC,YAAY,CAAC,IAAI,CAAC,GAAG,EAAE;YAC1B,IAAI,OAAO,IAAI,CAAC,OAAO,CAAC,qBAAqB,KAAK,UAAU,EAAE,CAAC;gBAC7D,IAAI,IAAI,CAAC,WAAW,EAAE,CAAC;oBACrB,aAAa,CAAC,IAAI,CAAC,WAAW,CAAC,CAAC;gBAClC,CAAC;gBACD,IAAI,CAAC,WAAW,GAAG,WAAW,CAC5B,GAAG,EAAE,eAAC,OAAA,MAAA,MAAA,IAAI,CAAC,OAAO,EAAC,qBAAqB,mDAAG,IAAI,CAAC,CAAA,EAAA,EAChD,IAAI,CAAC,OAAO,CAAC,uBAAuB,IAAI,KAAK,CAAC,CAAC;YACnD,CAAC;YACD,IAAI,CAAC,KAAK,CAAC,KAAK,EAAE,CAAC;QACrB,CAAC,CAAC,CAAC;IACL,CAAC;IAED,IAAI;QACF,IAAI,IAAI,CAAC,WAAW,EAAE,CAAC;YACrB,aAAa,CAAC,IAAI,CAAC,WAAW,CAAC,CAAC;QAClC,CAAC;QACD,IAAI,CAAC,KAAK,CAAC,KAAK,EAAE,CAAC;IACrB,CAAC;IAED,MAAM;QACJ,OAAO,IAAI,CAAC,YAAY,CAAC,IAAI,CAAC,GAAG,EAAE,CAAC,IAAI,CAAC,KAAK,CAAC,MAAM,EAAE,CAAC,CAAC;IAC3D,CAAC;IAED,KAAK,CAAC,OAAO;;QACX,IAAI,CAAC,IAAI,EAAE,CAAC;QACZ,IAAI,CAAC,KAAK,CAAC,KAAK,EAAE,CAAC;QACnB,MAAM,CAAA,MAAA,IAAI,CAAC,QAAQ,0CAAE,OAAO,CAAC,IAAI,CAAC,QAAQ,EAAE,IAAI,CAAC,CAAA,CAAC;IACpD,CAAC;CAEF"}

package/lib/downloader/multi.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"multi.d.ts","sourceRoot":"","sources":["../../src/downloader/multi.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAC,aAAa,EAAC,MAAM,kBAAkB,CAAC;AACpD,OAAO,EAAC,UAAU,EAAC,MAAM,kBAAkB,CAAC;AAC5C,OAAO,KAAK,EAAC,WAAW,EAAE,QAAQ,EAAC,MAAM,gBAAgB,CAAC;AAC1D,OAAO,KAAK,EAAC,qBAAqB,EAAC,MAAM,YAAY,CAAC;AACtD,OAAO,KAAK,EAAC,eAAe,EAAE,qBAAqB,EAAC,MAAM,eAAe,CAAC;AAG1E,OAAO,EAAC,kBAAkB,EAAC,MAAM,WAAW,CAAC;AAE7C,MAAM,WAAW,4BAA6B,SAAQ,qBAAqB;IACzE,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,OAAO,EAAE,MAAM,CAAC;CACjB;AAED,qBAAa,qBAAsB,SAAQ,kBAAkB;IAMlD,aAAa,EAAE,MAAM;IAE5B,OAAO,CAAC,cAAc,CAAC;IAPzB,OAAO,CAAC,KAAK,CAA6D;IAC1E,QAAQ,CAAC,IAAI,EAAE,OAAO,CAAC,IAAI,CAAC,CAAC;IAC7B,aAAa,EAAE,OAAO,CAAC,IAAI,CAAC,EAAE,CAAC;gBAGtB,aAAa,EAAE,MAAM,EAC5B,eAAe,CAAC,EAAE,OAAO,CAAC,4BAA4B,CAAC,EAC/C,cAAc,CAAC,EAAE,aAAa,YAAA;IAOxC,SAAS,CAAC,aAAa,CAAC,OAAO,EAAE,eAAe,GAAG,OAAO,CAAC,IAAI,CAAC;IA4BhE,IAAI,IAAI,IAAI,UAAU,CAAC,WAAW,EAAE,qBAAqB,CAAC,CAKzD;IAEK,0BAA0B,CAAC,GAAG,EAAE,QAAQ,GAAG,OAAO,CAAC,OAAO,GAAG,IAAI,CAAC;IAgDlE,OAAO,IAAI,OAAO,CAAC,IAAI,CAAC;CAO/B"}
+{"version":3,"file":"multi.d.ts","sourceRoot":"","sources":["../../src/downloader/multi.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAC,aAAa,EAAC,MAAM,kBAAkB,CAAC;AACpD,OAAO,EAAC,UAAU,EAAC,MAAM,kBAAkB,CAAC;AAC5C,OAAO,KAAK,EAAC,WAAW,EAAE,QAAQ,EAAC,MAAM,gBAAgB,CAAC;AAC1D,OAAO,KAAK,EAAC,qBAAqB,EAAC,MAAM,YAAY,CAAC;AACtD,OAAO,KAAK,EAAC,eAAe,EAAE,qBAAqB,EAAC,MAAM,eAAe,CAAC;AAE1E,OAAO,EAAC,kBAAkB,EAAC,MAAM,WAAW,CAAC;AAE7C,MAAM,WAAW,4BAA6B,SAAQ,qBAAqB;IACzE,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,OAAO,EAAE,MAAM,CAAC;CACjB;AAED,qBAAa,qBAAsB,SAAQ,kBAAkB;IAMlD,aAAa,EAAE,MAAM;IAE5B,OAAO,CAAC,cAAc,CAAC;IAPzB,OAAO,CAAC,KAAK,CAA6D;IAC1E,QAAQ,CAAC,IAAI,EAAE,OAAO,CAAC,IAAI,CAAC,CAAC;IAC7B,aAAa,EAAE,OAAO,CAAC,IAAI,CAAC,EAAE,CAAC;gBAGtB,aAAa,EAAE,MAAM,EAC5B,eAAe,CAAC,EAAE,OAAO,CAAC,4BAA4B,CAAC,EAC/C,cAAc,CAAC,EAAE,aAAa,YAAA;IAOxC,SAAS,CAAC,aAAa,CAAC,OAAO,EAAE,eAAe,GAAG,OAAO,CAAC,IAAI,CAAC;IA4BhE,IAAI,IAAI,IAAI,UAAU,CAAC,WAAW,EAAE,qBAAqB,CAAC,CAKzD;IAEK,0BAA0B,CAAC,GAAG,EAAE,QAAQ,GAAG,OAAO,CAAC,OAAO,GAAG,IAAI,CAAC;IA+ClE,OAAO,IAAI,OAAO,CAAC,IAAI,CAAC;CAO/B"}

package/lib/downloader/multi.js

@@ -1,6 +1,5 @@
 import path from 'node:path';
 import { WorkerPool } from './worker-pool.js';
-import { skip } from '../logger/logger.js';
 import { AbstractDownloader } from './main.js';
 export class MultiThreadDownloader extends AbstractDownloader {
     constructor(pathToOptions, overrideOptions, _workerFactory) {
@@ -29,7 +28,7 @@ export class MultiThreadDownloader extends AbstractDownloader {
             return this.addInitialResource(this.options.initialUrl);
         }
         else {
-            return this._initOptions.then(() => this.pipeline.init(this.pipeline, this));
+            return this.addInitialResource([]);
         }
     }
     get pool() {
@@ -44,7 +43,7 @@ export class MultiThreadDownloader extends AbstractDownloader {
         try {
             r = await this.pipeline.download(res);
             if (!r) {
-                skip.debug('discarded after download', res.url, res.rawUrl, res.refUrl);
+                await this.pipeline.notifyStatusChange(res, 'download');
                 return;
             }
         }
@@ -73,7 +72,7 @@ export class MultiThreadDownloader extends AbstractDownloader {
         }
         this.downloadedUrl.add(res.url);
         if (!msg) {
-            skip.info('discarded in post-processing', res.url, res.rawUrl, res.refUrl);
+            await this.pipeline.notifyStatusChange(res, 'processAfterDownload');
             return;
         }
         if (msg.error) {

package/lib/downloader/multi.js.map

@@ -1 +1 @@
-{"version":3,"file":"multi.js","sourceRoot":"","sources":["../../src/downloader/multi.ts"],"names":[],"mappings":"AAAA,OAAO,IAAI,MAAM,WAAW,CAAC;AAE7B,OAAO,EAAC,UAAU,EAAC,MAAM,kBAAkB,CAAC;AAK5C,OAAO,EAAC,IAAI,EAAC,MAAM,qBAAqB,CAAC;AACzC,OAAO,EAAC,kBAAkB,EAAC,MAAM,WAAW,CAAC;AAO7C,MAAM,OAAO,qBAAsB,SAAQ,kBAAkB;IAK3D,YACS,aAAqB,EAC5B,eAAuD,EAC/C,cAA8B;QAEtC,KAAK,CAAC,aAAa,EAAE,eAAe,CAAC,CAAC;QAJ/B,kBAAa,GAAb,aAAa,CAAQ;QAEpB,mBAAc,GAAd,cAAc,CAAgB;QAGtC,IAAI,CAAC,IAAI,GAAG,IAAI,CAAC,YAAY,CAAC;QAC9B,IAAI,CAAC,aAAa,GAAG,EAAE,CAAC;IAC1B,CAAC;IAES,aAAa,CAAC,OAAwB;QAC9C,IAAI,WAAW,GAAW,OAAO,CAAC,WAAW,CAAC;QAC9C,IAAI,OAAO,CAAC,WAAW,EAAE,CAAC;YACxB,WAAW,GAAG,IAAI,CAAC,GAAG,CAAC,OAAO,CAAC,WAAW,EAAE,WAAW,CAAC,CAAC;QAC3D,CAAC;QACD,IAAI,WAAW,GAAG,CAAC,EAAE,CAAC;YACpB,WAAW,GAAG,CAAC,CAAC;QAClB,CAAC;QACD,MAAM,eAAe,GAAG,OAAgD,CAAC;QACzE,IAAI,CAAC,KAAK,GAAG,IAAI,UAAU,CAAqC,WAAW;QACzE,0CAA0C;QAC1C,CAAA,eAAe,aAAf,eAAe,uBAAf,eAAe,CAAE,YAAY,KAAI,IAAI,CAAC,OAAO,CAAC,SAAS,EAAE,WAAW,CAAC,EACrE,EAAC,aAAa,EAAE,IAAI,CAAC,aAAa,EAAE,eAAe,EAAC,EACpD,CAAA,eAAe,aAAf,eAAe,uBAAf,eAAe,CAAE,OAAO,KAAI,CAAC,CAAC,EAC9B,IAAI,CAAC,cAAc,CACpB,CAAC;QACF,KAAK,MAAM,IAAI,IAAI,IAAI,CAAC,IAAI,CAAC,OAAO,EAAE,CAAC;YACrC,IAAI,CAAC,MAAM,CAAC,WAAW,CAAC,MAAM,EAC5B,QAAQ,CAAC,EAAE,CAAC,IAAI,CAAC,aAAa,CAAC,IAAI,CACjC,IAAI,CAAC,QAAQ,CAAC,OAAO,CAAC,IAAI,CAAC,QAAQ,EAAE,IAAI,EAAE,IAAI,EAAE,QAAQ,CAAC,CAAC,CAAC,CAAC;QACnE,CAAC;QACD,IAAI,IAAI,CAAC,OAAO,CAAC,UAAU,EAAE,CAAC;YAC5B,OAAO,IAAI,CAAC,kBAAkB,CAAC,IAAI,CAAC,OAAO,CAAC,UAAU,CAAC,CAAC;QAC1D,CAAC;aAAM,CAAC;YACN,OAAO,IAAI,CAAC,YAAY,CAAC,IAAI,CAAC,GAAG,EAAE,CAAC,IAAI,CAAC,QAAQ,CAAC,IAAI,CAAC,IAAI,CAAC,QAAQ,EAAE,IAAI,CAAC,CAAC,CAAC;QAC/E,CAAC;IACH,CAAC;IAED,IAAI,IAAI;QACN,IAAI,IAAI,CAAC,KAAK,EAAE,CAAC;YACf,OAAO,IAAI,CAAC,KAAK,CAAC;QACpB,CAAC;QACD,MAAM,IAAI,SAAS,CAAC,6CAA6C,CAAC,CAAC;IACrE,CAAC;IAED,KAAK,CAAC,0BAA0B,CAAC,GAAa;;QAC5C,IAAI,CAA0B,CAAC;QAC/B,IAAI,CAAC;YACH,CAAC,GAAG,MAAM,IAAI,CAAC,QAAS,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC;YACvC,IAAI,CAAC,CAAC,EAAE,CAAC;gBACP,IAAI,CAAC,KAAK,CAAC,0BAA0B,EAAE,GAAG,CAAC,GAAG,EAAE,GAAG,CAAC,MAAM,EAAE,GAAG,CAAC,MAAM,CAAC,CAAC;gBACxE,OAAO;YACT,CAAC;QACH,CAAC;QAAC,OAAO,CAAC,EAAE,CAAC;YACX,IAAI,CAAC,WAAW,CAAC,CAAC,EAAE,sBAAsB,EAAE,GAAG,CAAC,CAAC;YACjD,OAAO,KAAK,CAAC;QACf,CAAC;QACD,IAAI,GAAiC,CAAC;QACtC,IAAI,CAAC;YACH,IAAI,CAAC,WAAW,CAAC,MAAM,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,MAAM,CAAC,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC;gBACzD,CAAC,CAAC,IAAI,CAAC,UAAU,KAAK,CAAC;gBACvB,CAAC,CAAC,IAAI,CAAC,UAAU,KAAK,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC,UAAU;gBAC9C,CAAC,CAAC,IAAI,CAAC,MAAM,YAAY,WAAW,EAAE,CAAC;gBACvC,8DAA8D;gBAC9D,CAAC,CAAC,IAAI,GAAG,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC;gBACvB,GAAG,GAAG,MAAM,IAAI,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC;YAChD,CAAC;iBAAM,CAAC;gBACN,0BAA0B;gBAC1B,GAAG,GAAG,MAAM,IAAI,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC,CAAC,CAAC;YACtC,CAAC;QACH,CAAC;QAAC,OAAO,CAAC,EAAE,CAAC;YACX,IAAI,CAAC,WAAW,CAAC,CAAC,EAAE,+BAA+B,EAAE,GAAG,CAAC,CAAC;YAC1D,OAAO,KAAK,CAAC;QACf,CAAC;QACD,IAAI,CAAC,aAAa,CAAC,GAAG,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;QAChC,IAAI,CAAC,GAAG,EAAE,CAAC;YACT,IAAI,CAAC,IAAI,CAAC,8BAA8B,EACtC,GAAG,CAAC,GAAG,EAAE,GAAG,CAAC,MAAM,EAAE,GAAG,CAAC,MAAM,CAAC,CAAC;YACnC,OAAO;QACT,CAAC;QACD,IAAI,GAAG,CAAC,KAAK,EAAE,CAAC;YACd,IAAI,CAAC,WAAW,CAAC,GAAG,CAAC,KAAK,EAAE,cAAc,EAAE,GAAG,CAAC,CAAC;QACnD,CAAC;QACD,IAAI,MAAA,GAAG,CAAC,IAAI,0CAAE,MAAM,EAAE,CAAC;YACrB,MAAM,IAAI,GAAkB,GAAG,CAAC,IAAI,CAAC;YACrC,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,EAAE,CAAC,IAAI,CAAC,qBAAqB,CAAC,MAAM,CAAC,CAAC,CAAC;QAC7D,CAAC;QACD,IAAI,GAAG,CAAC,aAAa,EAAE,CAAC;YACtB,IAAI,CAAC,SAAS,CAAC,GAAG,CAAC,GAAG,CAAC,aAAa,CAAC,CAAC;QACxC,CAAC;IAEH,CAAC;IAED,KAAK,CAAC,OAAO;QACX,MAAM,KAAK,CAAC,OAAO,EAAE,CAAC;QACtB,MAAM,IAAI,CAAC,IAAI,CAAC,OAAO,EAAE,CAAC;QAC1B,MAAM,aAAa,GAAG,IAAI,CAAC,aAAa,CAAC;QACzC,IAAI,CAAC,aAAa,GAAG,EAAE,CAAC;QACxB,MAAM,OAAO,CAAC,GAAG,CAAC,aAAa,CAAC,CAAC;IACnC,CAAC;CACF"}
+{"version":3,"file":"multi.js","sourceRoot":"","sources":["../../src/downloader/multi.ts"],"names":[],"mappings":"AAAA,OAAO,IAAI,MAAM,WAAW,CAAC;AAE7B,OAAO,EAAC,UAAU,EAAC,MAAM,kBAAkB,CAAC;AAK5C,OAAO,EAAC,kBAAkB,EAAC,MAAM,WAAW,CAAC;AAO7C,MAAM,OAAO,qBAAsB,SAAQ,kBAAkB;IAK3D,YACS,aAAqB,EAC5B,eAAuD,EAC/C,cAA8B;QAEtC,KAAK,CAAC,aAAa,EAAE,eAAe,CAAC,CAAC;QAJ/B,kBAAa,GAAb,aAAa,CAAQ;QAEpB,mBAAc,GAAd,cAAc,CAAgB;QAGtC,IAAI,CAAC,IAAI,GAAG,IAAI,CAAC,YAAY,CAAC;QAC9B,IAAI,CAAC,aAAa,GAAG,EAAE,CAAC;IAC1B,CAAC;IAES,aAAa,CAAC,OAAwB;QAC9C,IAAI,WAAW,GAAW,OAAO,CAAC,WAAW,CAAC;QAC9C,IAAI,OAAO,CAAC,WAAW,EAAE,CAAC;YACxB,WAAW,GAAG,IAAI,CAAC,GAAG,CAAC,OAAO,CAAC,WAAW,EAAE,WAAW,CAAC,CAAC;QAC3D,CAAC;QACD,IAAI,WAAW,GAAG,CAAC,EAAE,CAAC;YACpB,WAAW,GAAG,CAAC,CAAC;QAClB,CAAC;QACD,MAAM,eAAe,GAAG,OAAgD,CAAC;QACzE,IAAI,CAAC,KAAK,GAAG,IAAI,UAAU,CAAqC,WAAW;QACzE,0CAA0C;QAC1C,CAAA,eAAe,aAAf,eAAe,uBAAf,eAAe,CAAE,YAAY,KAAI,IAAI,CAAC,OAAO,CAAC,SAAS,EAAE,WAAW,CAAC,EACrE,EAAC,aAAa,EAAE,IAAI,CAAC,aAAa,EAAE,eAAe,EAAC,EACpD,CAAA,eAAe,aAAf,eAAe,uBAAf,eAAe,CAAE,OAAO,KAAI,CAAC,CAAC,EAC9B,IAAI,CAAC,cAAc,CACpB,CAAC;QACF,KAAK,MAAM,IAAI,IAAI,IAAI,CAAC,IAAI,CAAC,OAAO,EAAE,CAAC;YACrC,IAAI,CAAC,MAAM,CAAC,WAAW,CAAC,MAAM,EAC5B,QAAQ,CAAC,EAAE,CAAC,IAAI,CAAC,aAAa,CAAC,IAAI,CACjC,IAAI,CAAC,QAAQ,CAAC,OAAO,CAAC,IAAI,CAAC,QAAQ,EAAE,IAAI,EAAE,IAAI,EAAE,QAAQ,CAAC,CAAC,CAAC,CAAC;QACnE,CAAC;QACD,IAAI,IAAI,CAAC,OAAO,CAAC,UAAU,EAAE,CAAC;YAC5B,OAAO,IAAI,CAAC,kBAAkB,CAAC,IAAI,CAAC,OAAO,CAAC,UAAU,CAAC,CAAC;QAC1D,CAAC;aAAM,CAAC;YACN,OAAO,IAAI,CAAC,kBAAkB,CAAC,EAAE,CAAC,CAAC;QACrC,CAAC;IACH,CAAC;IAED,IAAI,IAAI;QACN,IAAI,IAAI,CAAC,KAAK,EAAE,CAAC;YACf,OAAO,IAAI,CAAC,KAAK,CAAC;QACpB,CAAC;QACD,MAAM,IAAI,SAAS,CAAC,6CAA6C,CAAC,CAAC;IACrE,CAAC;IAED,KAAK,CAAC,0BAA0B,CAAC,GAAa;;QAC5C,IAAI,CAA0B,CAAC;QAC/B,IAAI,CAAC;YACH,CAAC,GAAG,MAAM,IAAI,CAAC,QAAS,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC;YACvC,IAAI,CAAC,CAAC,EAAE,CAAC;gBACP,MAAM,IAAI,CAAC,QAAQ,CAAC,kBAAkB,CAAC,GAAG,EAAE,UAAU,CAAC,CAAC;gBACxD,OAAO;YACT,CAAC;QACH,CAAC;QAAC,OAAO,CAAC,EAAE,CAAC;YACX,IAAI,CAAC,WAAW,CAAC,CAAC,EAAE,sBAAsB,EAAE,GAAG,CAAC,CAAC;YACjD,OAAO,KAAK,CAAC;QACf,CAAC;QACD,IAAI,GAAiC,CAAC;QACtC,IAAI,CAAC;YACH,IAAI,CAAC,WAAW,CAAC,MAAM,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,MAAM,CAAC,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC;gBACzD,CAAC,CAAC,IAAI,CAAC,UAAU,KAAK,CAAC;gBACvB,CAAC,CAAC,IAAI,CAAC,UAAU,KAAK,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC,UAAU;gBAC9C,CAAC,CAAC,IAAI,CAAC,MAAM,YAAY,WAAW,EAAE,CAAC;gBACvC,8DAA8D;gBAC9D,CAAC,CAAC,IAAI,GAAG,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC;gBACvB,GAAG,GAAG,MAAM,IAAI,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC;YAChD,CAAC;iBAAM,CAAC;gBACN,0BAA0B;gBAC1B,GAAG,GAAG,MAAM,IAAI,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC,CAAC,CAAC;YACtC,CAAC;QACH,CAAC;QAAC,OAAO,CAAC,EAAE,CAAC;YACX,IAAI,CAAC,WAAW,CAAC,CAAC,EAAE,+BAA+B,EAAE,GAAG,CAAC,CAAC;YAC1D,OAAO,KAAK,CAAC;QACf,CAAC;QACD,IAAI,CAAC,aAAa,CAAC,GAAG,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;QAChC,IAAI,CAAC,GAAG,EAAE,CAAC;YACT,MAAM,IAAI,CAAC,QAAQ,CAAC,kBAAkB,CAAC,GAAG,EAAE,sBAAsB,CAAC,CAAC;YACpE,OAAO;QACT,CAAC;QACD,IAAI,GAAG,CAAC,KAAK,EAAE,CAAC;YACd,IAAI,CAAC,WAAW,CAAC,GAAG,CAAC,KAAK,EAAE,cAAc,EAAE,GAAG,CAAC,CAAC;QACnD,CAAC;QACD,IAAI,MAAA,GAAG,CAAC,IAAI,0CAAE,MAAM,EAAE,CAAC;YACrB,MAAM,IAAI,GAAkB,GAAG,CAAC,IAAI,CAAC;YACrC,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,EAAE,CAAC,IAAI,CAAC,qBAAqB,CAAC,MAAM,CAAC,CAAC,CAAC;QAC7D,CAAC;QACD,IAAI,GAAG,CAAC,aAAa,EAAE,CAAC;YACtB,IAAI,CAAC,SAAS,CAAC,GAAG,CAAC,GAAG,CAAC,aAAa,CAAC,CAAC;QACxC,CAAC;IAEH,CAAC;IAED,KAAK,CAAC,OAAO;QACX,MAAM,KAAK,CAAC,OAAO,EAAE,CAAC;QACtB,MAAM,IAAI,CAAC,IAAI,CAAC,OAAO,EAAE,CAAC;QAC1B,MAAM,aAAa,GAAG,IAAI,CAAC,aAAa,CAAC;QACzC,IAAI,CAAC,aAAa,GAAG,EAAE,CAAC;QACxB,MAAM,OAAO,CAAC,GAAG,CAAC,aAAa,CAAC,CAAC;IACnC,CAAC;CACF"}

package/lib/downloader/pipeline-executor-impl.d.ts

@@ -1,6 +1,6 @@
 import type { StaticDownloadOptions } from '../options.js';
-import type { Resource, ResourceEncoding, ResourceType } from '../resource.js';
-import type { DownloadResource, ProcessingLifeCycle, RequestOptions, SubmitResourceFunc } from '../life-cycle/types.js';
+import type { RawResource, Resource, ResourceEncoding, ResourceType } from '../resource.js';
+import type { DownloadResource, InitSubmitFunc, ProcessingLifeCycle, RequestOptions, ResourceStatus, SubmitResourceFunc } from '../life-cycle/types.js';
 import type { PipelineExecutor } from '../life-cycle/pipeline-executor.js';
 import type { Cheerio } from '../types.js';
 import type { DownloaderWithMeta } from './types.js';
@@ -13,7 +13,7 @@ export declare class PipelineExecutorImpl implements PipelineExecutor {
     requestOptions: RequestOptions;
     options: StaticDownloadOptions;
     constructor(lifeCycle: ProcessingLifeCycle, requestOptions: RequestOptions, options: StaticDownloadOptions);
-    init(pipeline: PipelineExecutor, downloader?: DownloaderWithMeta): Promise<void>;
+    init(pipeline: PipelineExecutor, downloader?: DownloaderWithMeta, submit?: InitSubmitFunc): Promise<void>;
     createAndProcessResource(rawUrl: string, defaultType: ResourceType, depth: number | void | null, element: Cheerio | null, parent: Resource): Promise<Resource | void>;
     linkRedirect(url: string, element: Cheerio | null, parent: Resource | null): Promise<string | void>;
     detectResourceType(url: string, type: ResourceType, element: Cheerio | null, parent: Resource | null): Promise<ResourceType | void>;
@@ -29,5 +29,8 @@ export declare class PipelineExecutorImpl implements PipelineExecutor {
     processAfterDownload(res: DownloadResource, submit: SubmitResourceFunc, options?: StaticDownloadOptions): Promise<DownloadResource | void>;
     saveToDisk(res: DownloadResource, options?: StaticDownloadOptions): Promise<DownloadResource | void>;
     dispose(pipeline: PipelineExecutor, downloader: DownloaderWithMeta, workerInfo?: WorkerInfo, workerExitCode?: number): Promise<void>;
+    notifyStatusChange(res: Resource | RawResource, status: ResourceStatus): Promise<void>;
+    private _checkExistingResource;
+    private _getExistingFileMtime;
 }
 //# sourceMappingURL=pipeline-executor-impl.d.ts.map

package/lib/downloader/pipeline-executor-impl.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"pipeline-executor-impl.d.ts","sourceRoot":"","sources":["../../src/downloader/pipeline-executor-impl.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAC,qBAAqB,EAAC,MAAM,eAAe,CAAC;AACzD,OAAO,KAAK,EAEV,QAAQ,EACR,gBAAgB,EAChB,YAAY,EACb,MAAM,gBAAgB,CAAC;AACxB,OAAO,KAAK,EACV,gBAAgB,EAChB,mBAAmB,EACnB,cAAc,EACd,kBAAkB,EACnB,MAAM,wBAAwB,CAAC;AAEhC,OAAO,KAAK,EAAC,gBAAgB,EAAC,MAAM,oCAAoC,CAAC;AACzE,OAAO,KAAK,EAAC,OAAO,EAAC,MAAM,aAAa,CAAC;AACzC,OAAO,KAAK,EAAC,kBAAkB,EAAC,MAAM,YAAY,CAAC;AACnD,OAAO,KAAK,EAAC,UAAU,EAAC,MAAM,kBAAkB,CAAC;AAEjD;;GAEG;AACH,qBAAa,oBAAqB,YAAW,gBAAgB;IACxC,SAAS,EAAE,mBAAmB;IAC9B,cAAc,EAAE,cAAc;IAC9B,OAAO,EAAE,qBAAqB;gBAF9B,SAAS,EAAE,mBAAmB,EAC9B,cAAc,EAAE,cAAc,EAC9B,OAAO,EAAE,qBAAqB;IAG3C,IAAI,CACR,QAAQ,EAAE,gBAAgB,EAC1B,UAAU,CAAC,EAAE,kBAAkB,GAC9B,OAAO,CAAC,IAAI,CAAC;IAOV,wBAAwB,CAC5B,MAAM,EAAE,MAAM,EACd,WAAW,EAAE,YAAY,EACzB,KAAK,EAAE,MAAM,GAAG,IAAI,GAAG,IAAI,EAC3B,OAAO,EAAE,OAAO,GAAG,IAAI,EACvB,MAAM,EAAE,QAAQ,GACf,OAAO,CAAC,QAAQ,GAAG,IAAI,CAAC;IAiBrB,YAAY,CAChB,GAAG,EAAE,MAAM,EACX,OAAO,EAAE,OAAO,GAAG,IAAI,EACvB,MAAM,EAAE,QAAQ,GAAG,IAAI,GACtB,OAAO,CAAC,MAAM,GAAG,IAAI,CAAC;IAYnB,kBAAkB,CACtB,GAAG,EAAE,MAAM,EACX,IAAI,EAAE,YAAY,EAClB,OAAO,EAAE,OAAO,GAAG,IAAI,EACvB,MAAM,EAAE,QAAQ,GAAG,IAAI,GACtB,OAAO,CAAC,YAAY,GAAG,IAAI,CAAC;IAc/B,cAAc,CACZ,IAAI,EAAE,YAAY,EAClB,KAAK,EAAE,MAAM,EACb,GAAG,EAAE,MAAM,EACX,MAAM,EAAE,MAAM,EACd,SAAS,CAAC,EAAE,MAAM,EAClB,QAAQ,CAAC,EAAE,gBAAgB,EAC3B,WAAW,CAAC,EAAE,MAAM,EACpB,OAAO,CAAC,EAAE,YAAY,GACrB,QAAQ;IAkBL,qBAAqB,CACzB,GAAG,EAAE,QAAQ,EACb,OAAO,EAAE,OAAO,GAAG,IAAI,EACvB,MAAM,EAAE,QAAQ,GAAG,IAAI,EACvB,OAAO,CAAC,EAAE,qBAAqB,GAC9B,OAAO,CAAC,QAAQ,GAAG,IAAI,CAAC;IAgBrB,QAAQ,CACZ,GAAG,EAAE,QAAQ,EACb,cAAc,CAAC,EAAE,cAAc,EAC/B,OAAO,CAAC,EAAE,qBAAqB,GAC9B,OAAO,CAAC,gBAAgB,GAAG,IAAI,CAAC;IA0BnC;;;;;OAKG;IACG,oBAAoB,CACxB,GAAG,EAAE,gBAAgB,EACrB,MAAM,EAAE,kBAAkB,EAC1B,OAAO,CAAC,EAAE,qBAAqB,GAC9B,OAAO,CAAC,gBAAgB,GAAG,IAAI,CAAC;IAe7B,UAAU,CACd,GAAG,EAAE,gBAAgB,EACrB,OAAO,CAAC,EAAE,qBAAqB,GAC9B,OAAO,CAAC,gBAAgB,GAAG,IAAI,CAAC;IAiB7B,OAAO,CACX,QAAQ,EAAE,gBAAgB,EAC1B,UAAU,EAAE,kBAAkB,EAC9B,UAAU,CAAC,EAAE,UAAU,EACvB,cAAc,CAAC,EAAE,MAAM,GACtB,OAAO,CAAC,IAAI,CAAC;CAOjB"}
+{"version":3,"file":"pipeline-executor-impl.d.ts","sourceRoot":"","sources":["../../src/downloader/pipeline-executor-impl.ts"],"names":[],"mappings":"AAGA,OAAO,KAAK,EAAC,qBAAqB,EAAC,MAAM,eAAe,CAAC;AACzD,OAAO,KAAK,EAEV,WAAW,EACX,QAAQ,EACR,gBAAgB,EAChB,YAAY,EACb,MAAM,gBAAgB,CAAC;AACxB,OAAO,KAAK,EACV,gBAAgB,EAGhB,cAAc,EACd,mBAAmB,EACnB,cAAc,EACd,cAAc,EACd,kBAAkB,EACnB,MAAM,wBAAwB,CAAC;AAEhC,OAAO,KAAK,EAAC,gBAAgB,EAAC,MAAM,oCAAoC,CAAC;AACzE,OAAO,KAAK,EAAC,OAAO,EAAC,MAAM,aAAa,CAAC;AACzC,OAAO,KAAK,EAAC,kBAAkB,EAAC,MAAM,YAAY,CAAC;AACnD,OAAO,KAAK,EAAC,UAAU,EAAC,MAAM,kBAAkB,CAAC;AAEjD;;GAEG;AACH,qBAAa,oBAAqB,YAAW,gBAAgB;IACxC,SAAS,EAAE,mBAAmB;IAC9B,cAAc,EAAE,cAAc;IAC9B,OAAO,EAAE,qBAAqB;gBAF9B,SAAS,EAAE,mBAAmB,EAC9B,cAAc,EAAE,cAAc,EAC9B,OAAO,EAAE,qBAAqB;IAG3C,IAAI,CACR,QAAQ,EAAE,gBAAgB,EAC1B,UAAU,CAAC,EAAE,kBAAkB,EAC/B,MAAM,CAAC,EAAE,cAAc,GACtB,OAAO,CAAC,IAAI,CAAC;IAOV,wBAAwB,CAC5B,MAAM,EAAE,MAAM,EACd,WAAW,EAAE,YAAY,EACzB,KAAK,EAAE,MAAM,GAAG,IAAI,GAAG,IAAI,EAC3B,OAAO,EAAE,OAAO,GAAG,IAAI,EACvB,MAAM,EAAE,QAAQ,GACf,OAAO,CAAC,QAAQ,GAAG,IAAI,CAAC;IAiBrB,YAAY,CAChB,GAAG,EAAE,MAAM,EACX,OAAO,EAAE,OAAO,GAAG,IAAI,EACvB,MAAM,EAAE,QAAQ,GAAG,IAAI,GACtB,OAAO,CAAC,MAAM,GAAG,IAAI,CAAC;IAYnB,kBAAkB,CACtB,GAAG,EAAE,MAAM,EACX,IAAI,EAAE,YAAY,EAClB,OAAO,EAAE,OAAO,GAAG,IAAI,EACvB,MAAM,EAAE,QAAQ,GAAG,IAAI,GACtB,OAAO,CAAC,YAAY,GAAG,IAAI,CAAC;IAc/B,cAAc,CACZ,IAAI,EAAE,YAAY,EAClB,KAAK,EAAE,MAAM,EACb,GAAG,EAAE,MAAM,EACX,MAAM,EAAE,MAAM,EACd,SAAS,CAAC,EAAE,MAAM,EAClB,QAAQ,CAAC,EAAE,gBAAgB,EAC3B,WAAW,CAAC,EAAE,MAAM,EACpB,OAAO,CAAC,EAAE,YAAY,GACrB,QAAQ;IAkBL,qBAAqB,CACzB,GAAG,EAAE,QAAQ,EACb,OAAO,EAAE,OAAO,GAAG,IAAI,EACvB,MAAM,EAAE,QAAQ,GAAG,IAAI,EACvB,OAAO,CAAC,EAAE,qBAAqB,GAC9B,OAAO,CAAC,QAAQ,GAAG,IAAI,CAAC;IAgBrB,QAAQ,CACZ,GAAG,EAAE,QAAQ,EACb,cAAc,CAAC,EAAE,cAAc,EAC/B,OAAO,CAAC,EAAE,qBAAqB,GAC9B,OAAO,CAAC,gBAAgB,GAAG,IAAI,CAAC;IA0CnC;;;;;OAKG;IACG,oBAAoB,CACxB,GAAG,EAAE,gBAAgB,EACrB,MAAM,EAAE,kBAAkB,EAC1B,OAAO,CAAC,EAAE,qBAAqB,GAC9B,OAAO,CAAC,gBAAgB,GAAG,IAAI,CAAC;IAe7B,UAAU,CACd,GAAG,EAAE,gBAAgB,EACrB,OAAO,CAAC,EAAE,qBAAqB,GAC9B,OAAO,CAAC,gBAAgB,GAAG,IAAI,CAAC;IAwC7B,OAAO,CACX,QAAQ,EAAE,gBAAgB,EAC1B,UAAU,EAAE,kBAAkB,EAC9B,UAAU,CAAC,EAAE,UAAU,EACvB,cAAc,CAAC,EAAE,MAAM,GACtB,OAAO,CAAC,IAAI,CAAC;IAOV,kBAAkB,CACtB,GAAG,EAAE,QAAQ,GAAG,WAAW,EAC3B,MAAM,EAAE,cAAc,GACrB,OAAO,CAAC,IAAI,CAAC;IAYhB,OAAO,CAAC,sBAAsB;IAqB9B,OAAO,CAAC,qBAAqB;CAY9B"}

package/lib/downloader/pipeline-executor-impl.js

@@ -1,3 +1,5 @@
+import path from 'node:path';
+import { existsSync, statSync } from 'node:fs';
 /**
  * Pipeline executor
  */
@@ -7,11 +9,11 @@ export class PipelineExecutorImpl {
         this.requestOptions = requestOptions;
         this.options = options;
     }
-    async init(pipeline, downloader) {
+    async init(pipeline, downloader, submit) {
         if (!this.lifeCycle.init)
             return;
         for (const init of this.lifeCycle.init) {
-            await init(pipeline, downloader);
+            await init(pipeline, downloader, submit);
         }
     }
     async createAndProcessResource(rawUrl, defaultType, depth, element, parent) {
@@ -91,6 +93,22 @@ export class PipelineExecutorImpl {
         if (!options) {
             options = this.options;
         }
+        if (this.lifeCycle.existingResource) {
+            const action = this._checkExistingResource(res, 'download');
+            if (action === 'skip') {
+                res.shouldBeDiscardedFromDownload = true;
+                return undefined;
+            }
+            if (action === 'ifModifiedSince') {
+                const mtime = this._getExistingFileMtime(res);
+                if (mtime) {
+                    requestOptions = Object.assign({}, requestOptions);
+                    requestOptions.headers = Object.assign({}, requestOptions.headers, {
+                        'if-modified-since': mtime
+                    });
+                }
+            }
+        }
         let downloadedResource = res;
         for (const download of this.lifeCycle.download) {
             if ((downloadedResource = await download(downloadedResource, requestOptions, options, this))
@@ -125,9 +143,31 @@ export class PipelineExecutorImpl {
         return downloadedResource;
     }
     async saveToDisk(res, options) {
+        var _a, _b, _c;
         if (!options) {
             options = this.options;
         }
+        if (this.lifeCycle.existingResource) {
+            const action = this._checkExistingResource(res, 'saveToDisk');
+            if (action === 'skip' || action === 'skipSave') {
+                return undefined;
+            }
+            if (action === 'ifModifiedSince') {
+                const remoteLastMod = (_b = (_a = res.meta) === null || _a === void 0 ? void 0 : _a.headers) === null || _b === void 0 ? void 0 : _b['last-modified'];
+                if (remoteLastMod) {
+                    const localPath = path.join((_c = res.localRoot) !== null && _c !== void 0 ? _c : this.options.localRoot, decodeURI(res.savePath));
+                    try {
+                        const localMtime = statSync(localPath).mtime;
+                        if (new Date(remoteLastMod) <= localMtime) {
+                            return undefined;
+                        }
+                    }
+                    catch (_d) {
+                        // file removed between check and stat, proceed with save
+                    }
+                }
+            }
+        }
         let downloadedResource = res;
         for (const saveToDisk of this.lifeCycle.saveToDisk) {
             if ((downloadedResource = await saveToDisk(downloadedResource, options, this))
@@ -146,5 +186,49 @@ export class PipelineExecutorImpl {
             await dispose(pipeline, downloader, workerInfo, workerExitCode);
         }
     }
+    async notifyStatusChange(res, status) {
+        var _a;
+        if (!((_a = this.lifeCycle.statusChange) === null || _a === void 0 ? void 0 : _a.length))
+            return;
+        for (const listener of this.lifeCycle.statusChange) {
+            try {
+                const r = listener(res, status, this.options, this);
+                if (r)
+                    await r;
+            }
+            catch (_b) {
+                // swallow
+            }
+        }
+    }
+    _checkExistingResource(res, stage) {
+        var _a;
+        const localPath = path.join((_a = res.localRoot) !== null && _a !== void 0 ? _a : this.options.localRoot, decodeURI(res.savePath));
+        if (!existsSync(localPath))
+            return undefined;
+        let stat;
+        try {
+            stat = statSync(localPath);
+        }
+        catch (_b) {
+            // TOCTOU: file deleted between existsSync and statSync
+            return undefined;
+        }
+        if (!stat.isFile())
+            return undefined;
+        return this.lifeCycle.existingResource({
+            res, stage, localPath, stat, options: this.options
+        });
+    }
+    _getExistingFileMtime(res) {
+        var _a;
+        const localPath = path.join((_a = res.localRoot) !== null && _a !== void 0 ? _a : this.options.localRoot, decodeURI(res.savePath));
+        try {
+            return statSync(localPath).mtime.toUTCString();
+        }
+        catch (_b) {
+            return undefined;
+        }
+    }
 }
 //# sourceMappingURL=pipeline-executor-impl.js.map