@crawlee/basic 3.0.0-alpha.2 → 3.0.0-alpha.3

package/dist/LICENSE.md

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "{}"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2018 Apify Technologies s.r.o.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

package/dist/index.d.ts

@@ -0,0 +1,4 @@
+export * from '@crawlee/core';
+export * from './internals/basic-crawler';
+export * from './internals/constants';
+//# sourceMappingURL=index.d.ts.map

package/dist/index.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,cAAc,eAAe,CAAC;AAC9B,cAAc,2BAA2B,CAAC;AAC1C,cAAc,uBAAuB,CAAC"}

package/dist/index.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":";;;AAAA,wDAA8B;AAC9B,oEAA0C;AAC1C,gEAAsC"}

package/dist/internals/basic-crawler.d.ts

@@ -0,0 +1,379 @@
+import { Log } from '@apify/log';
+import { AutoscaledPool, AutoscaledPoolOptions, EnqueueLinksOptions, CrawlerHandleFailedRequestInput, ProxyInfo, QueueOperationInfo, Request, RequestList, RequestQueue, Session, SessionPool, SessionPoolOptions, Statistics, type CrawlingContext, RequestOptions, RequestQueueOperationOptions, Configuration, EventManager, FinalStatistics } from '@crawlee/core';
+import { Awaitable } from '@crawlee/utils';
+export interface BasicCrawlerCrawlingContext extends CrawlingContext {
+    crawler: BasicCrawler;
+    enqueueLinks: (options: BasicCrawlerEnqueueLinksOptions) => Promise<QueueOperationInfo[]>;
+}
+export interface BasicCrawlerHandleFailedRequestInput extends CrawlerHandleFailedRequestInput {
+    crawler: BasicCrawler;
+}
+/** @internal */
+export declare type BasicCrawlerEnqueueLinksOptions = Omit<EnqueueLinksOptions, 'requestQueue'>;
+export declare type RequestHandler<Context extends CrawlingContext = BasicCrawlerCrawlingContext> = (inputs: Context) => Awaitable<void>;
+export declare type FailedRequestHandler<Inputs extends CrawlerHandleFailedRequestInput = BasicCrawlerHandleFailedRequestInput> = (inputs: Inputs) => Awaitable<void>;
+export interface BasicCrawlerOptions<Context extends CrawlingContext = BasicCrawlerCrawlingContext, ErrorContext extends CrawlerHandleFailedRequestInput = BasicCrawlerHandleFailedRequestInput> {
+    /**
+     * User-provided function that performs the logic of the crawler. It is called for each URL to crawl.
+     *
+     * The function receives the following object as an argument:
+     * ```
+     * {
+     *     request: Request,
+     *     session: Session,
+     *     crawler: BasicCrawler,
+     * }
+     * ```
+     * where the {@link Request} instance represents the URL to crawl.
+     *
+     * The function must return a promise, which is then awaited by the crawler.
+     *
+     * If the function throws an exception, the crawler will try to re-crawl the
+     * request later, up to `option.maxRequestRetries` times.
+     * If all the retries fail, the crawler calls the function
+     * provided to the `handleFailedRequestFunction` parameter.
+     * To make this work, you should **always**
+     * let your function throw exceptions rather than catch them.
+     * The exceptions are logged to the request using the
+     * {@link Request.pushErrorMessage} function.
+     */
+    requestHandler: RequestHandler<Context>;
+    /**
+     * User-provided function that performs the logic of the crawler. It is called for each URL to crawl.
+     *
+     * The function receives the following object as an argument:
+     * ```
+     * {
+     *     request: Request,
+     *     session: Session,
+     *     crawler: BasicCrawler,
+     * }
+     * ```
+     * where the {@link Request} instance represents the URL to crawl.
+     *
+     * The function must return a promise, which is then awaited by the crawler.
+     *
+     * If the function throws an exception, the crawler will try to re-crawl the
+     * request later, up to `option.maxRequestRetries` times.
+     * If all the retries fail, the crawler calls the function
+     * provided to the `handleFailedRequestFunction` parameter.
+     * To make this work, you should **always**
+     * let your function throw exceptions rather than catch them.
+     * The exceptions are logged to the request using the
+     * {@link Request.pushErrorMessage} function.
+     *
+     * @deprecated `handleRequestFunction` has been renamed to `requestHandler` and will be removed in a future version.
+     */
+    handleRequestFunction?: RequestHandler<Context>;
+    /**
+     * Static list of URLs to be processed.
+     * Either `requestList` or `requestQueue` option must be provided (or both).
+     */
+    requestList?: RequestList;
+    /**
+     * Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
+     * Either `requestList` or `requestQueue` option must be provided (or both).
+     */
+    requestQueue?: RequestQueue;
+    /**
+     * Timeout in which the function passed as `requestHandler` needs to finish, in seconds.
+     * @default 60
+     */
+    requestHandlerTimeoutSecs?: number;
+    /**
+     * Timeout in which the function passed as `requestHandler` needs to finish, in seconds.
+     * @default 60
+     * @deprecated `handleRequestTimeoutSecs` has been renamed to `requestHandlerTimeoutSecs` and will be removed in a future version.
+     */
+    handleRequestTimeoutSecs?: number;
+    /**
+     * A function to handle requests that failed more than `option.maxRequestRetries` times.
+     *
+     * The function receives the following object as an argument:
+     * ```
+     * {
+     *     request: Request,
+     *     error: Error,
+     *     session: Session,
+     *     crawler: BasicCrawler,
+     * }
+     * ```
+     * where the {@link Request} instance corresponds to the failed request, and the `Error` instance
+     * represents the last error thrown during processing of the request.
+     *
+     * See
+     * [source code](https://github.com/apify/apify-js/blob/master/src/crawlers/basic_crawler.js#L11)
+     * for the default implementation of this function.
+     */
+    failedRequestHandler?: FailedRequestHandler<ErrorContext>;
+    /**
+     * A function to handle requests that failed more than `option.maxRequestRetries` times.
+     *
+     * The function receives the following object as an argument:
+     * ```
+     * {
+     *     request: Request,
+     *     error: Error,
+     *     session: Session,
+     *     crawler: BasicCrawler,
+     * }
+     * ```
+     * where the {@link Request} instance corresponds to the failed request, and the `Error` instance
+     * represents the last error thrown during processing of the request.
+     *
+     * See
+     * [source code](https://github.com/apify/apify-js/blob/master/src/crawlers/basic_crawler.js#L11)
+     * for the default implementation of this function.
+     *
+     * @deprecated `handleFailedRequestFunction` has been renamed to `failedRequestHandler` and will be removed in a future version.
+     */
+    handleFailedRequestFunction?: FailedRequestHandler<ErrorContext>;
+    /**
+     * Indicates how many times the request is retried if {@link requestHandler} or {@link handlePageFunction} fails.
+     * @default 3
+     */
+    maxRequestRetries?: number;
+    /**
+     * Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached.
+     * Always set this value in order to prevent infinite loops in misconfigured crawlers.
+     * Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.
+     */
+    maxRequestsPerCrawl?: number;
+    /**
+     * Custom options passed to the underlying {@link AutoscaledPool} constructor.
+     * Note that the `runTaskFunction` and `isTaskReadyFunction` options
+     * are provided by the crawler and cannot be overridden.
+     * However, you can provide a custom implementation of `isFinishedFunction`.
+     */
+    autoscaledPoolOptions?: AutoscaledPoolOptions;
+    /**
+     * Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
+     *
+     * *WARNING:* If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.
+     * If you're not sure, just keep the default value and the concurrency will scale up automatically.
+     */
+    minConcurrency?: number;
+    /**
+     * Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
+     */
+    maxConcurrency?: number;
+    /**
+     * Basic crawler will initialize the  {@link SessionPool} with the corresponding `sessionPoolOptions`.
+     * The session instance will be than available in the `handleRequestFunction`.
+     */
+    useSessionPool?: boolean;
+    /**
+     * The configuration options for {@link SessionPool} to use.
+     */
+    sessionPoolOptions?: SessionPoolOptions;
+    /** @internal */
+    log?: Log;
+}
+/**
+ * Provides a simple framework for parallel crawling of web pages.
+ * The URLs to crawl are fed either from a static list of URLs
+ * or from a dynamic queue of URLs enabling recursive crawling of websites.
+ *
+ * `BasicCrawler` is a low-level tool that requires the user to implement the page
+ * download and data extraction functionality themselves.
+ * If you want a crawler that already facilitates this functionality,
+ * please consider using {@link CheerioCrawler}, {@link PuppeteerCrawler} or {@link PlaywrightCrawler}.
+ *
+ * `BasicCrawler` invokes the user-provided {@link BasicCrawlerOptions.requestHandler}
+ * for each {@link Request} object, which represents a single URL to crawl.
+ * The {@link Request} objects are fed from the {@link RequestList} or the {@link RequestQueue}
+ * instances provided by the {@link BasicCrawlerOptions.requestList} or {@link BasicCrawlerOptions.requestQueue}
+ * constructor options, respectively.
+ *
+ * If both {@link BasicCrawlerOptions.requestList} and {@link BasicCrawlerOptions.requestQueue} options are used,
+ * the instance first processes URLs from the {@link RequestList} and automatically enqueues all of them
+ * to {@link RequestQueue} before it starts their processing. This ensures that a single URL is not crawled multiple times.
+ *
+ * The crawler finishes if there are no more {@link Request} objects to crawl.
+ *
+ * New requests are only dispatched when there is enough free CPU and memory available,
+ * using the functionality provided by the {@link AutoscaledPool} class.
+ * All {@link AutoscaledPool} configuration options can be passed to the `autoscaledPoolOptions`
+ * parameter of the `BasicCrawler` constructor. For user convenience, the `minConcurrency` and `maxConcurrency`
+ * {@link AutoscaledPool} options are available directly in the `BasicCrawler` constructor.
+ *
+ * **Example usage:**
+ *
+ * ```javascript
+ * const { gotScraping } = require('got-scraping');
+ *
+ * // Prepare a list of URLs to crawl
+ * const requestList = new RequestList({
+ *   sources: [
+ *       { url: 'http://www.example.com/page-1' },
+ *       { url: 'http://www.example.com/page-2' },
+ *   ],
+ * });
+ * await requestList.initialize();
+ *
+ * // Crawl the URLs
+ * const crawler = new BasicCrawler({
+ *     requestList,
+ *     handleRequestFunction: async ({ request }) => {
+ *         // 'request' contains an instance of the Request class
+ *         // Here we simply fetch the HTML of the page and store it to a dataset
+ *         const { body } = await gotScraping({
+ *             url: request.url,
+ *             method: request.method,
+ *             body: request.payload,
+ *             headers: request.headers,
+ *         });
+ *
+ *         await Actor.pushData({
+ *             url: request.url,
+ *             html: body,
+ *         })
+ *     },
+ * });
+ *
+ * await crawler.run();
+ * ```
+ * @category Crawlers
+ */
+export declare class BasicCrawler<Context extends CrawlingContext = BasicCrawlerCrawlingContext, ErrorContext extends CrawlerHandleFailedRequestInput = BasicCrawlerHandleFailedRequestInput> {
+    readonly config: Configuration;
+    /**
+     * Static list of URLs to be processed.
+     */
+    readonly stats: Statistics;
+    /**
+     * A reference to the underlying {@link RequestList} class that manages the crawler's {@link Request}s.
+     * Either `requestList` or `requestQueue` option must be provided (or both).
+     * Only available if used by the crawler.
+     */
+    requestList?: RequestList;
+    /**
+     * Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
+     * A reference to the underlying {@link RequestQueue} class that manages the crawler's {@link Request}s.
+     * Either `requestList` or `requestQueue` option must be provided (or both).
+     * Only available if used by the crawler.
+     */
+    requestQueue?: RequestQueue;
+    /**
+     * A reference to the underlying {@link SessionPool} class that manages the crawler's {@link Session}s.
+     * Only available if used by the crawler.
+     */
+    sessionPool?: SessionPool;
+    /**
+     * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.
+     * Note that this property is only initialized after calling the {@link BasicCrawler.run} function.
+     * You can use it to change the concurrency settings on the fly,
+     * to pause the crawler by calling {@link AutoscaledPool.pause}
+     * or to abort it by calling {@link AutoscaledPool.abort}.
+     */
+    autoscaledPool?: AutoscaledPool;
+    protected log: Log;
+    protected requestHandler: RequestHandler<Context>;
+    protected failedRequestHandler?: FailedRequestHandler<ErrorContext>;
+    protected requestHandlerTimeoutMillis: number;
+    protected internalTimeoutMillis: number;
+    protected maxRequestRetries: number;
+    protected handledRequestsCount: number;
+    protected sessionPoolOptions: SessionPoolOptions;
+    protected useSessionPool: boolean;
+    protected crawlingContexts: Map<string, Context>;
+    protected autoscaledPoolOptions: AutoscaledPoolOptions;
+    protected events: EventManager;
+    protected static optionsShape: {
+        requestList: import("ow").ObjectPredicate<object> & import("ow").BasePredicate<object | undefined>;
+        requestQueue: import("ow").ObjectPredicate<object> & import("ow").BasePredicate<object | undefined>;
+        requestHandler: import("ow").Predicate<Function> & import("ow").BasePredicate<Function | undefined>;
+        handleRequestFunction: import("ow").Predicate<Function> & import("ow").BasePredicate<Function | undefined>;
+        requestHandlerTimeoutSecs: import("ow").NumberPredicate & import("ow").BasePredicate<number | undefined>;
+        handleRequestTimeoutSecs: import("ow").NumberPredicate & import("ow").BasePredicate<number | undefined>;
+        failedRequestHandler: import("ow").Predicate<Function> & import("ow").BasePredicate<Function | undefined>;
+        handleFailedRequestFunction: import("ow").Predicate<Function> & import("ow").BasePredicate<Function | undefined>;
+        maxRequestRetries: import("ow").NumberPredicate & import("ow").BasePredicate<number | undefined>;
+        maxRequestsPerCrawl: import("ow").NumberPredicate & import("ow").BasePredicate<number | undefined>;
+        autoscaledPoolOptions: import("ow").ObjectPredicate<object> & import("ow").BasePredicate<object | undefined>;
+        sessionPoolOptions: import("ow").ObjectPredicate<object> & import("ow").BasePredicate<object | undefined>;
+        useSessionPool: import("ow").BooleanPredicate & import("ow").BasePredicate<boolean | undefined>;
+        minConcurrency: import("ow").NumberPredicate & import("ow").BasePredicate<number | undefined>;
+        maxConcurrency: import("ow").NumberPredicate & import("ow").BasePredicate<number | undefined>;
+        log: import("ow").ObjectPredicate<object> & import("ow").BasePredicate<object | undefined>;
+    };
+    /**
+     * All `BasicCrawler` parameters are passed via an options object.
+     */
+    constructor(options: BasicCrawlerOptions<Context, ErrorContext>, config?: Configuration);
+    /**
+     * Runs the crawler. Returns a promise that gets resolved once all the requests are processed.
+     */
+    run(): Promise<FinalStatistics>;
+    getRequestQueue(): Promise<RequestQueue>;
+    /**
+     * Adds requests to be processed by the crawler
+     * @param requests The requests to add
+     * @param options Options for the request queue
+     */
+    addRequests(requests: (string | Request | RequestOptions)[], options?: RequestQueueOperationOptions): Promise<import("packages/core/dist/storages/storage").BatchAddRequestsResult>;
+    protected _init(): Promise<void>;
+    protected _runRequestHandler(crawlingContext: Context): Promise<void>;
+    protected _pauseOnMigration(): Promise<void>;
+    /**
+     * Fetches request from either RequestList or RequestQueue. If request comes from a RequestList
+     * and RequestQueue is present then enqueues it to the queue first.
+     */
+    protected _fetchNextRequest(): Promise<Request | null>;
+    /**
+     * Wrapper around requestHandler that fetches requests from RequestList/RequestQueue
+     * then retries them in a case of an error, etc.
+     */
+    protected _runTaskFunction(): Promise<void>;
+    /**
+     * Run async callback with given timeout and retry.
+     * @ignore
+     */
+    protected _timeoutAndRetry(handler: () => Promise<unknown>, timeout: number, error: Error | string, maxRetries?: number, retried?: number): Promise<void>;
+    /**
+     * Returns true if either RequestList or RequestQueue have a request ready for processing.
+     */
+    protected _isTaskReadyFunction(): Promise<boolean>;
+    /**
+     * Returns true if both RequestList and RequestQueue have all requests finished.
+     */
+    protected _defaultIsFinishedFunction(): Promise<boolean>;
+    /**
+     * Handles errors thrown by user provided handleRequestFunction()
+     */
+    protected _requestFunctionErrorHandler(error: Error, crawlingContext: Context, source: RequestList | RequestQueue): Promise<void>;
+    protected _handleFailedRequestHandler(crawlingContext: ErrorContext): Promise<void>;
+    /**
+     * Updates handledRequestsCount from possibly stored counts,
+     * usually after worker migration. Since one of the stores
+     * needs to have priority when both are present,
+     * it is the request queue, because generally, the request
+     * list will first be dumped into the queue and then left
+     * empty.
+     */
+    protected _loadHandledRequestCount(): Promise<void>;
+    protected _executeHooks<HookLike extends (...args: any[]) => Awaitable<void>>(hooks: HookLike[], ...args: Parameters<HookLike>): Promise<void>;
+    /**
+     * Function for cleaning up after all request are processed.
+     * @ignore
+     */
+    teardown(): Promise<void>;
+    protected _handlePropertyNameChange<New, Old>({ newProperty, newName, oldProperty, oldName, propertyKey, allowUndefined, }: HandlePropertyNameChangeData<New, Old>): void;
+}
+/** @internal */
+export declare function basicCrawlerEnqueueLinks(options: BasicCrawlerEnqueueLinksOptions, requestQueue?: RequestQueue): Promise<import("packages/core/dist/storages/storage").BatchAddRequestsResult>;
+export interface CreateContextOptions {
+    request: Request;
+    session?: Session;
+    proxyInfo?: ProxyInfo;
+}
+interface HandlePropertyNameChangeData<New, Old> {
+    oldProperty?: Old;
+    newProperty?: New;
+    oldName: string;
+    newName: string;
+    propertyKey: string;
+    allowUndefined?: boolean;
+}
+export {};
+//# sourceMappingURL=basic-crawler.d.ts.map

package/dist/internals/basic-crawler.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"basic-crawler.d.ts","sourceRoot":"","sources":["../../src/internals/basic-crawler.ts"],"names":[],"mappings":"AAAA,OAAmB,EAAE,GAAG,EAAE,MAAM,YAAY,CAAC;AAG7C,OAAO,EACH,cAAc,EACd,qBAAqB,EACrB,mBAAmB,EACnB,+BAA+B,EAE/B,SAAS,EACT,kBAAkB,EAClB,OAAO,EACP,WAAW,EACX,YAAY,EACZ,OAAO,EACP,WAAW,EACX,kBAAkB,EAClB,UAAU,EAEV,KAAK,eAAe,EACpB,cAAc,EACd,4BAA4B,EAE5B,aAAa,EACb,YAAY,EAEZ,eAAe,EAClB,MAAM,eAAe,CAAC;AACvB,OAAO,EAAE,SAAS,EAAE,MAAM,gBAAgB,CAAC;AAG3C,MAAM,WAAW,2BAA4B,SAAQ,eAAe;IAChE,OAAO,EAAE,YAAY,CAAC;IACtB,YAAY,EAAE,CAAC,OAAO,EAAE,+BAA+B,KAAK,OAAO,CAAC,kBAAkB,EAAE,CAAC,CAAC;CAC7F;AAED,MAAM,WAAW,oCAAqC,SAAQ,+BAA+B;IACzF,OAAO,EAAE,YAAY,CAAC;CACzB;AAED,gBAAgB;AAChB,oBAAY,+BAA+B,GAAG,IAAI,CAAC,mBAAmB,EAAE,cAAc,CAAC,CAAA;AAavF,oBAAY,cAAc,CAAC,OAAO,SAAS,eAAe,GAAG,2BAA2B,IAAI,CAAC,MAAM,EAAE,OAAO,KAAK,SAAS,CAAC,IAAI,CAAC,CAAC;AAEjI,oBAAY,oBAAoB,CAAC,MAAM,SAAS,+BAA+B,GAAG,oCAAoC,IAAI,CAAC,MAAM,EAAE,MAAM,KAAK,SAAS,CAAC,IAAI,CAAC,CAAC;AAE9J,MAAM,WAAW,mBAAmB,CAChC,OAAO,SAAS,eAAe,GAAG,2BAA2B,EAC7D,YAAY,SAAS,+BAA+B,GAAG,oCAAoC;IAE3F;;;;;;;;;;;;;;;;;;;;;;;OAuBG;IACH,cAAc,EAAE,cAAc,CAAC,OAAO,CAAC,CAAC;IAExC;;;;;;;;;;;;;;;;;;;;;;;;;OAyBG;IACH,qBAAqB,CAAC,EAAE,cAAc,CAAC,OAAO,CAAC,CAAC;IAEhD;;;OAGG;IACH,WAAW,CAAC,EAAE,WAAW,CAAC;IAE1B;;;OAGG;IACH,YAAY,CAAC,EAAE,YAAY,CAAC;IAE5B;;;OAGG;IACH,yBAAyB,CAAC,EAAE,MAAM,CAAC;IAEnC;;;;OAIG;IACH,wBAAwB,CAAC,EAAE,MAAM,CAAC;IAElC;;;;;;;;;;;;;;;;;;OAkBG;IACH,oBAAoB,CAAC,EAAE,oBAAoB,CAAC,YAAY,CAAC,CAAC;IAE1D;;;;;;;;;;;;;;;;;;;;OAoBG;IACH,2BAA2B,CAAC,EAAE,oBAAoB,CAAC,YAAY,CAAC,CAAC;IAEjE;;;OAGG;IACH,iBAAiB,CAAC,EAAE,MAAM,CAAC;IAE3B;;;;OAIG;IACH,mBAAmB,CAAC,EAAE,MAAM,CAAC;IAE7B;;;;;OAKG;IACH,qBAAqB,CAAC,EAAE,qBAAqB,CAAC;IAE9C;;;;;OAKG;IACH,cAAc,CAAC,EAAE,MAAM,CAAC;IAExB;;OAEG;IACH,cAAc,CAAC,EAAE,MAAM,CAAC;IAExB;;;OAGG;IACH,cAAc,CAAC,EAAE,OAAO,CAAC;IAEzB;;OAEG;IACH,kBAAkB,CAAC,EAAE,kBAAkB,CAAC;IAExC,gBAAgB;IAChB,GAAG,CAAC,EAAE,GAAG,CAAC;CACb;AAED;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAiEG;AACH,qBAAa,YAAY,CACrB,OAAO,SAAS,eAAe,GAAG,2BAA2B,EAC7D,YAAY,SAAS,+BAA+B,GAAG,oCAAoC;IAmF1B,QAAQ,CAAC,MAAM;IAjFhF;;OAEG;IACH,QAAQ,CAAC,KAAK,EAAE,UAAU,CAAC;IAE3B;;;;OAIG;IACH,WAAW,CAAC,EAAE,WAAW,CAAC;IAE1B;;;;;OAKG;IACH,YAAY,CAAC,EAAE,YAAY,CAAC;IAE5B;;;OAGG;IACH,WAAW,CAAC,EAAE,WAAW,CAAC;IAE1B;;;;;;OAMG;IACH,cAAc,CAAC,EAAE,cAAc,CAAC;IAEhC,SAAS,CAAC,GAAG,EAAE,GAAG,CAAC;IACnB,SAAS,CAAC,cAAc,EAAG,cAAc,CAAC,OAAO,CAAC,CAAC;IACnD,SAAS,CAAC,oBAAoB,CAAC,EAAE,oBAAoB,CAAC,YAAY,CAAC,CAAC;IACpE,SAAS,CAAC,2BAA2B,EAAG,MAAM,CAAC;IAC/C,SAAS,CAAC,qBAAqB,EAAE,MAAM,CAAC;IACxC,SAAS,CAAC,iBAAiB,EAAE,MAAM,CAAC;IACpC,SAAS,CAAC,oBAAoB,EAAE,MAAM,CAAC;IACvC,SAAS,CAAC,kBAAkB,EAAE,kBAAkB,CAAC;IACjD,SAAS,CAAC,cAAc,EAAE,OAAO,CAAC;IAClC,SAAS,CAAC,gBAAgB,uBAA8B;IACxD,SAAS,CAAC,qBAAqB,EAAE,qBAAqB,CAAC;IACvD,SAAS,CAAC,MAAM,EAAE,YAAY,CAAC;IAE/B,SAAS,CAAC,MAAM,CAAC,YAAY;;;;;;;;;;;;;;;;;MA4B3B;IAEF;;OAEG;gBACS,OAAO,EAAE,mBAAmB,CAAC,OAAO,EAAE,YAAY,CAAC,EAAW,MAAM,gBAAkC;IA+IlH;;OAEG;IACG,GAAG,IAAI,OAAO,CAAC,eAAe,CAAC;IAuB/B,eAAe;IAMrB;;;;OAIG;IACG,WAAW,CAAC,QAAQ,EAAE,CAAC,MAAM,GAAG,OAAO,GAAG,cAAc,CAAC,EAAE,EAAE,OAAO,GAAE,4BAAiC;cAgB7F,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC;cAetB,kBAAkB,CAAC,eAAe,EAAE,OAAO,GAAG,OAAO,CAAC,IAAI,CAAC;cAI3D,iBAAiB;IAqCjC;;;OAGG;cACa,iBAAiB;IAmBjC;;;OAGG;cACa,gBAAgB;IAyFhC;;;OAGG;cACa,gBAAgB,CAAC,OAAO,EAAE,MAAM,OAAO,CAAC,OAAO,CAAC,EAAE,OAAO,EAAE,MAAM,EAAE,KAAK,EAAE,KAAK,GAAG,MAAM,EAAE,UAAU,SAAI,EAAE,OAAO,SAAI,GAAG,OAAO,CAAC,IAAI,CAAC;IAarJ;;OAEG;cACa,oBAAoB;IASpC;;OAEG;cACa,0BAA0B;IAY1C;;OAEG;cACa,4BAA4B,CACxC,KAAK,EAAE,KAAK,EACZ,eAAe,EAAE,OAAO,EACxB,MAAM,EAAE,WAAW,GAAG,YAAY,GACnC,OAAO,CAAC,IAAI,CAAC;cA8BA,2BAA2B,CAAC,eAAe,EAAE,YAAY,GAAG,OAAO,CAAC,IAAI,CAAC;IAazF;;;;;;;OAOG;cACa,wBAAwB,IAAI,OAAO,CAAC,IAAI,CAAC;cAQzC,aAAa,CAAC,QAAQ,SAAS,CAAC,GAAG,IAAI,EAAE,GAAG,EAAE,KAAK,SAAS,CAAC,IAAI,CAAC,EAAE,KAAK,EAAE,QAAQ,EAAE,EAAE,GAAG,IAAI,EAAE,UAAU,CAAC,QAAQ,CAAC;IAQpI;;;OAGG;IACG,QAAQ,IAAI,OAAO,CAAC,IAAI,CAAC;IAM/B,SAAS,CAAC,yBAAyB,CAAC,GAAG,EAAE,GAAG,EAAE,EAC1C,WAAW,EACX,OAAO,EACP,WAAW,EACX,OAAO,EACP,WAAW,EACX,cAAsB,GACzB,EAAE,4BAA4B,CAAC,GAAG,EAAE,GAAG,CAAC;CAyB5C;AAED,gBAAgB;AAChB,wBAAsB,wBAAwB,CAAC,OAAO,EAAE,+BAA+B,EAAE,YAAY,CAAC,EAAE,YAAY,iFAMnH;AAED,MAAM,WAAW,oBAAoB;IACjC,OAAO,EAAE,OAAO,CAAC;IACjB,OAAO,CAAC,EAAE,OAAO,CAAC;IAClB,SAAS,CAAC,EAAE,SAAS,CAAC;CACzB;AAED,UAAU,4BAA4B,CAAC,GAAG,EAAE,GAAG;IAC3C,WAAW,CAAC,EAAE,GAAG,CAAC;IAClB,WAAW,CAAC,EAAE,GAAG,CAAC;IAClB,OAAO,EAAE,MAAM,CAAC;IAChB,OAAO,EAAE,MAAM,CAAC;IAChB,WAAW,EAAE,MAAM,CAAC;IACpB,cAAc,CAAC,EAAE,OAAO,CAAC;CAC5B"}