@gby/got-scraping 4.1.3

package/README.md

@@ -0,0 +1,164 @@
+> # ⚠️⚠️⚠️ `got-scraping` is EOL ⚠️⚠️⚠️
+>
+> After many years of development, we decided to deprecate the `got-scraping` package.
+> The package will no longer receive updates or support.
+>
+> For new projects, we recommend using [`impit`](https://github.com/apify/impit). `impit` is a modern, powerful, and flexible HTTP client with `fetch` API based on Rust's `reqwest` library. It provides a similar feature set to `got-scraping`, including browser-like request headers, proxy support, and more.
+>
+
+
+## Got Scraping
+
+Got Scraping is a small but powerful [`got` extension](https://github.com/sindresorhus/got) with the purpose of sending browser-like requests out of the box. This is very essential in the web scraping industry to blend in with the website traffic.
+
+## Installation
+
+```
+$ npm install got-scraping
+```
+
+# The module is now ESM only
+
+This means you have to import it by using an `import` expression, or the `import()` method. You can do so by either migrating your project to ESM, or importing `got-scraping` in an async context
+
+```diff
+-const { gotScraping } = require('got-scraping');
++import { gotScraping } from 'got-scraping';
+```
+
+If you cannot migrate to ESM, here's an example of how to import it in an async context:
+
+```javascript
+let gotScraping;
+
+async function fetchWithGotScraping(url) {
+    gotScraping ??= (await import('got-scraping')).gotScraping;
+
+    return gotScraping.get(url);
+}
+```
+
+**Note:**
+> - Node.js >=16 is required due to instability of HTTP/2 support in lower versions.
+
+## API
+
+Got scraping package is built using the [`got.extend(...)`](https://github.com/sindresorhus/got/blob/main/documentation/10-instances.md) functionality, therefore it supports all the features Got has.
+
+Interested what's [under the hood](#under-the-hood)?
+
+```javascript
+import { gotScraping } from 'got-scraping';
+
+gotScraping
+    .get('https://apify.com')
+    .then( ({ body }) => console.log(body));
+```
+
+### options
+
+#### `proxyUrl`
+
+Type: **`string`**
+
+URL of the HTTP or HTTPS based proxy. HTTP/2 proxies are supported as well.
+
+```javascript
+import { gotScraping } from 'got-scraping';
+
+gotScraping
+    .get({
+        url: 'https://apify.com',
+        proxyUrl: 'http://usernamed:password@myproxy.com:1234',
+    })
+    .then(({ body }) => console.log(body));
+```
+
+#### `useHeaderGenerator`
+
+Type: **`boolean`**\
+Default: **`true`**
+
+Whether to use the generation of the browser-like headers.
+
+#### `headerGeneratorOptions`
+
+See the [`HeaderGeneratorOptions`](https://github.com/apify/fingerprint-suite/tree/master/packages/header-generator#headergeneratoroptions) docs.
+
+```javascript
+const response = await gotScraping({
+    url: 'https://api.apify.com/v2/browser-info',
+    headerGeneratorOptions:{
+        browsers: [
+            {
+                name: 'chrome',
+                minVersion: 87,
+                maxVersion: 89
+            }
+        ],
+        devices: ['desktop'],
+        locales: ['de-DE', 'en-US'],
+        operatingSystems: ['windows', 'linux'],
+    }
+});
+```
+
+#### `sessionToken`
+
+A non-primitive unique object which describes the current session. By default, it's `undefined`, so new headers will be generated every time. Headers generated with the same `sessionToken` never change.
+
+## Under the hood
+
+Thanks to the included [`header-generator`](https://github.com/apify/fingerprint-suite/tree/master/packages/header-generator) package, you can choose various browsers from different operating systems and devices. It generates all the headers automatically so you can focus on the important stuff instead.
+
+Yet another goal is to simplify the usage of proxies. Just pass the `proxyUrl` option and you are set. Got Scraping automatically detects the HTTP protocol that the proxy server supports. After the connection is established, it does another ALPN negotiation for the end server. Once that is complete, Got Scraping can proceed with HTTP requests.
+
+Using the same HTTP version that browsers do is important as well. Most modern browsers use HTTP/2, so Got Scraping is making a use of it too. Fortunately, this is already supported by Got - it automatically handles [ALPN protocol negotiation](https://en.wikipedia.org/wiki/Application-Layer_Protocol_Negotiation) to select the best available protocol.
+
+HTTP/1.1 headers are always automatically formatted in [`Pascal-Case`](https://pl.wikipedia.org/wiki/PascalCase). However, there is an exception: [`x-`](https://datatracker.ietf.org/doc/html/rfc7231#section-8.3.1) headers are not modified in *any* way.
+
+By default, Got Scraping will use an insecure HTTP parser, which allows to access websites with non-spec-compliant web servers.
+
+Last but not least, Got Scraping comes with updated TLS configuration. Some websites make a fingerprint of it and compare it with real browsers. While Node.js doesn't support OpenSSL 3 yet, the current configuration still should work flawlessly.
+
+To get more detailed information about the implementation, please refer to the [source code](https://github.com/apify/got-scraping/blob/master/src/index.ts).
+
+## Tips
+
+This package can only generate all the standard attributes. You might want to add the [`referer` header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Referer) if necessary. Please bear in mind that these headers are made for GET requests for HTML documents. If you want to make POST requests or GET requests for any other content type, you should alter these headers according to your needs. You can do so by passing a headers option or writing a custom [Got handler](https://github.com/sindresorhus/got/blob/main/documentation/10-instances.md).
+
+This package should provide a solid start for your browser request emulation process. All websites are built differently, and some of them might require some additional special care.
+
+### Overriding request headers
+
+```javascript
+const response = await gotScraping({
+    url: 'https://apify.com/',
+    headers: {
+        'user-agent': 'test',
+    },
+});
+```
+
+For more advanced usage please refer to the [Got documentation](https://github.com/sindresorhus/got/#documentation).
+
+### JSON mode
+
+You can parse JSON with this package too, but please bear in mind that the request header generation is done specifically for `HTML` content type. You might want to alter the generated headers to match the browser ones.
+
+```javascript
+const response = await gotScraping({
+    responseType: 'json',
+    url: 'https://api.apify.com/v2/browser-info',
+});
+```
+
+### Error recovery
+
+This section covers possible errors that might happen due to different site implementations.
+
+```
+RequestError: Client network socket disconnected before secure TLS connection was established
+```
+
+The error above can be a result of the server not supporting the provided TLS setings. Try changing the ciphers parameter to either `undefined` or a custom value.

package/dist/index.d.ts

@@ -0,0 +1,284 @@
+// @ts-ignore Patch needed for ES20xx compatibility while this module uses Node16/NodeNext resolutions
+import * as got from 'got';
+// @ts-ignore Patch needed for ES20xx compatibility while this module uses Node16/NodeNext resolutions
+import { Options, OptionsInit as OptionsInit$1, Agents, CancelableRequest, Response, Request, ExtendOptions, HTTPAlias, PaginationOptions, PaginateData, Got } from 'got';
+// @ts-ignore Patch needed for ES20xx compatibility while this module uses Node16/NodeNext resolutions
+export * from 'got';
+// @ts-ignore Patch needed for ES20xx compatibility while this module uses Node16/NodeNext resolutions
+export { OptionsInit as GotOptionsInit } from 'got';
+import http, { Agent, ClientRequest, ClientRequestArgs, AgentOptions } from 'node:http';
+import https from 'node:https';
+import { HeaderGenerator } from 'header-generator';
+import { NetConnectOpts } from 'node:net';
+import { Duplex } from 'node:stream';
+import { URL as URL$1 } from 'node:url';
+
+/**
+ * @see https://github.com/nodejs/node/blob/533cafcf7e3ab72e98a2478bc69aedfdf06d3a5e/lib/_http_client.js#L129-L162
+ * @see https://github.com/nodejs/node/blob/533cafcf7e3ab72e98a2478bc69aedfdf06d3a5e/lib/_http_client.js#L234-L246
+ * @see https://github.com/nodejs/node/blob/533cafcf7e3ab72e98a2478bc69aedfdf06d3a5e/lib/_http_client.js#L304-L305
+ * Wraps an existing Agent instance,
+ * so there's no need to replace `agent.addRequest`.
+ */
+declare class WrappedAgent<T extends Agent> implements Agent {
+    agent: T;
+    constructor(agent: T);
+    addRequest(request: ClientRequest, options: ClientRequestArgs): void;
+    get keepAlive(): boolean;
+    get maxSockets(): Agent['maxSockets'];
+    get options(): AgentOptions;
+    get defaultPort(): number;
+    get protocol(): string;
+    destroy(): void;
+    get maxFreeSockets(): Agent['maxFreeSockets'];
+    get maxTotalSockets(): Agent['maxTotalSockets'];
+    get freeSockets(): Agent['freeSockets'];
+    get sockets(): Agent['sockets'];
+    get requests(): Agent['requests'];
+    on(eventName: string | symbol, listener: (...args: any[]) => void): this;
+    once(eventName: string | symbol, listener: (...args: any[]) => void): this;
+    off(eventName: string | symbol, listener: (...args: any[]) => void): this;
+    addListener(eventName: string | symbol, listener: (...args: any[]) => void): this;
+    removeListener(eventName: string | symbol, listener: (...args: any[]) => void): this;
+    removeAllListeners(eventName?: string | symbol): this;
+    setMaxListeners(n: number): this;
+    getMaxListeners(): number;
+    listeners(eventName: Parameters<Agent['listeners']>[0]): ReturnType<Agent['listeners']>;
+    rawListeners(eventName: Parameters<Agent['rawListeners']>[0]): ReturnType<Agent['rawListeners']>;
+    emit(eventName: string | symbol, ...args: any[]): boolean;
+    eventNames(): (string | symbol)[];
+    listenerCount(eventName: string | symbol): number;
+    prependListener(eventName: string | symbol, listener: (...args: any[]) => void): this;
+    prependOnceListener(eventName: string | symbol, listener: (...args: any[]) => void): this;
+    createConnection(options: NetConnectOpts, callback?: (err: Error | null, stream: Duplex) => void): Duplex;
+    keepSocketAlive(socket: Duplex): void;
+    reuseSocket(socket: Duplex, request: ClientRequest): void;
+    getName(options?: any): string;
+}
+
+/**
+ * Transforms the casing of the headers to Pascal-Case.
+ */
+declare class TransformHeadersAgent<T extends Agent> extends WrappedAgent<T> {
+    /**
+     * Transforms the request via header normalization.
+     */
+    transformRequest(request: ClientRequest, { sortHeaders }: {
+        sortHeaders: boolean;
+    }): void;
+    addRequest(request: ClientRequest, options: ClientRequestArgs): void;
+    toPascalCase(header: string): string;
+}
+
+declare function browserHeadersHook(options: Options): Promise<void>;
+
+declare function customOptionsHook(raw: OptionsInit$1, options: Options): void;
+
+declare function http2Hook(options: Options): void;
+
+declare function insecureParserHook(options: Options): void;
+
+declare function optionsValidationHandler(options: unknown): void;
+
+declare function proxyHook(options: Options): Promise<void>;
+declare function getAgents(parsedProxyUrl: URL$1, rejectUnauthorized: boolean): Promise<Agents>;
+
+declare function tlsHook(options: Options): void;
+
+interface Context {
+    proxyUrl?: string;
+    headerGeneratorOptions?: Record<string, unknown>;
+    useHeaderGenerator?: boolean;
+    headerGenerator?: {
+        getHeaders: (options: Record<string, unknown>) => Record<string, string>;
+    };
+    insecureHTTPParser?: boolean;
+    sessionToken?: object;
+    /** @private */
+    sessionData?: unknown;
+    /** @private */
+    resolveProtocol?: (data: unknown) => {
+        alpnProtocol: string;
+    } | Promise<{
+        alpnProtocol: string;
+    }>;
+}
+type OptionsInit = OptionsInit$1 & Context;
+
+type Except<ObjectType, KeysType extends keyof ObjectType> = Pick<ObjectType, Exclude<keyof ObjectType, KeysType>>;
+type Merge<FirstType, SecondType> = Except<FirstType, Extract<keyof FirstType, keyof SecondType>> & SecondType;
+type ExtendedGotRequestFunction = {
+    (url: string | URL, options?: ExtendedOptionsOfTextResponseBody): CancelableRequest<Response<string>>;
+    <T>(url: string | URL, options?: ExtendedOptionsOfJSONResponseBody): CancelableRequest<Response<T>>;
+    (url: string | URL, options?: ExtendedOptionsOfBufferResponseBody): CancelableRequest<Response<Buffer>>;
+    (url: string | URL, options?: ExtendedOptionsOfUnknownResponseBody): CancelableRequest<Response>;
+    (options: ExtendedOptionsOfTextResponseBody): CancelableRequest<Response<string>>;
+    <T>(options: ExtendedOptionsOfJSONResponseBody): CancelableRequest<Response<T>>;
+    (options: ExtendedOptionsOfBufferResponseBody): CancelableRequest<Response<Buffer>>;
+    (options: ExtendedOptionsOfUnknownResponseBody): CancelableRequest<Response>;
+    (url: string | URL, options?: (Merge<ExtendedOptionsOfTextResponseBody, ResponseBodyOnly>)): CancelableRequest<string>;
+    <T>(url: string | URL, options?: (Merge<ExtendedOptionsOfJSONResponseBody, ResponseBodyOnly>)): CancelableRequest<T>;
+    (url: string | URL, options?: (Merge<ExtendedOptionsOfBufferResponseBody, ResponseBodyOnly>)): CancelableRequest<Buffer>;
+    (options: (Merge<ExtendedOptionsOfTextResponseBody, ResponseBodyOnly>)): CancelableRequest<string>;
+    <T>(options: (Merge<ExtendedOptionsOfJSONResponseBody, ResponseBodyOnly>)): CancelableRequest<T>;
+    (options: (Merge<ExtendedOptionsOfBufferResponseBody, ResponseBodyOnly>)): CancelableRequest<Buffer>;
+    (url: string | URL, options?: Merge<OptionsInit, {
+        isStream: true;
+    }>): Request;
+    (options: Merge<OptionsInit, {
+        isStream: true;
+    }>): Request;
+    (url: string | URL, options?: OptionsInit): CancelableRequest | Request;
+    (options: OptionsInit): CancelableRequest | Request;
+    (url: undefined, options: undefined, defaults: Options): CancelableRequest | Request;
+};
+type ExtendedOptionsOfTextResponseBody = Merge<OptionsInit, {
+    isStream?: false;
+    resolveBodyOnly?: false;
+    responseType?: 'text';
+}>;
+type ExtendedOptionsOfJSONResponseBody = Merge<OptionsInit, {
+    isStream?: false;
+    resolveBodyOnly?: false;
+    responseType?: 'json';
+}>;
+type ExtendedOptionsOfBufferResponseBody = Merge<OptionsInit, {
+    isStream?: false;
+    resolveBodyOnly?: false;
+    responseType: 'buffer';
+}>;
+type ExtendedOptionsOfUnknownResponseBody = Merge<OptionsInit, {
+    isStream?: false;
+    resolveBodyOnly?: false;
+}>;
+type ResponseBodyOnly = {
+    resolveBodyOnly: true;
+};
+type ExtendedGotStreamFunction = ((url?: string | URL, options?: Merge<OptionsInit, {
+    isStream?: true;
+}>) => Request) & ((options?: Merge<OptionsInit, {
+    isStream?: true;
+}>) => Request);
+type ExtendedExtendOptions = ExtendOptions & OptionsInit;
+type ExtendedGotStream = ExtendedGotStreamFunction & Record<HTTPAlias, ExtendedGotStreamFunction>;
+type ExtendedPaginationOptions<ElementType, BodyType> = PaginationOptions<ElementType, BodyType> & {
+    paginate?: (data: PaginateData<BodyType, ElementType>) => OptionsInit | false;
+};
+type ExtendedOptionsWithPagination<T = unknown, R = unknown> = Merge<OptionsInit, {
+    pagination?: ExtendedPaginationOptions<T, R>;
+}>;
+type ExtendedGotPaginate = {
+    /**
+    Same as `GotPaginate.each`.
+    */
+    <T, R = unknown>(url: string | URL, options?: ExtendedOptionsWithPagination<T, R>): AsyncIterableIterator<T>;
+    /**
+    Same as `GotPaginate.each`.
+    */
+    <T, R = unknown>(options?: ExtendedOptionsWithPagination<T, R>): AsyncIterableIterator<T>;
+    /**
+    Returns an async iterator.
+
+    See pagination.options for more pagination options.
+
+    @example
+    ```
+    import { gotScraping } from 'got-scraping';
+
+    const countLimit = 10;
+
+    const pagination = gotScraping.paginate('https://api.github.com/repos/sindresorhus/got/commits', {
+        pagination: { countLimit }
+    });
+
+    console.log(`Printing latest ${countLimit} Got commits (newest to oldest):`);
+
+    for await (const commitData of pagination) {
+        console.log(commitData.commit.message);
+    }
+    ```
+    */
+    each: (<T, R = unknown>(url: string | URL, options?: ExtendedOptionsWithPagination<T, R>) => AsyncIterableIterator<T>) & (<T, R = unknown>(options?: ExtendedOptionsWithPagination<T, R>) => AsyncIterableIterator<T>);
+    /**
+    Returns a Promise for an array of all results.
+
+    See pagination.options for more pagination options.
+
+    @example
+    ```
+    import { gotScraping } from 'got-scraping';
+
+    const countLimit = 10;
+
+    const results = await gotScraping.paginate.all('https://api.github.com/repos/sindresorhus/got/commits', {
+        pagination: { countLimit }
+    });
+
+    console.log(`Printing latest ${countLimit} Got commits (newest to oldest):`);
+    console.log(results);
+    ```
+    */
+    all: (<T, R = unknown>(url: string | URL, options?: ExtendedOptionsWithPagination<T, R>) => Promise<T[]>) & (<T, R = unknown>(options?: ExtendedOptionsWithPagination<T, R>) => Promise<T[]>);
+};
+type GotScraping = {
+    stream: ExtendedGotStream;
+    paginate: ExtendedGotPaginate;
+    defaults: Got['defaults'];
+    extend: (...instancesOrOptions: Array<GotScraping | ExtendedExtendOptions>) => GotScraping;
+} & Record<HTTPAlias, ExtendedGotRequestFunction> & ExtendedGotRequestFunction;
+
+/**
+ * Create the options for the Got Scraping instance.
+ * @returns got 的扩展选项 ExtendOptions
+ */
+declare function createGotScrapingOptions(): {
+    handlers: got.HandlerFunction[];
+    mutableDefaults: boolean;
+    http2: boolean;
+    https: {
+        rejectUnauthorized: boolean;
+    };
+    throwHttpErrors: boolean;
+    timeout: {
+        request: number;
+    };
+    retry: {
+        limit: number;
+    };
+    headers: {
+        'user-agent': undefined;
+    };
+    context: {
+        headerGenerator: HeaderGenerator;
+        useHeaderGenerator: boolean;
+        insecureHTTPParser: boolean;
+    };
+    agent: {
+        http: TransformHeadersAgent<http.Agent>;
+        https: TransformHeadersAgent<https.Agent>;
+    };
+    hooks: {
+        init: (typeof customOptionsHook)[];
+        beforeRequest: (typeof insecureParserHook)[];
+        beforeRedirect: got.BeforeRedirectHook[];
+    };
+};
+declare const gotScraping: GotScraping;
+
+declare const hooks: {
+    init: (typeof customOptionsHook)[];
+    beforeRequest: (typeof insecureParserHook)[];
+    beforeRedirect: got.BeforeRedirectHook[];
+    fixDecompress: got.HandlerFunction;
+    insecureParserHook: typeof insecureParserHook;
+    sessionDataHook: (options: Options) => void;
+    http2Hook: typeof http2Hook;
+    proxyHook: typeof proxyHook;
+    browserHeadersHook: typeof browserHeadersHook;
+    tlsHook: typeof tlsHook;
+    optionsValidationHandler: typeof optionsValidationHandler;
+    customOptionsHook: typeof customOptionsHook;
+    refererHook: got.BeforeRedirectHook;
+};
+
+export { Context, ExtendedExtendOptions, ExtendedGotPaginate, ExtendedGotRequestFunction, ExtendedGotStream, ExtendedGotStreamFunction, ExtendedOptionsOfBufferResponseBody, ExtendedOptionsOfJSONResponseBody, ExtendedOptionsOfTextResponseBody, ExtendedOptionsOfUnknownResponseBody, ExtendedOptionsWithPagination, ExtendedPaginationOptions, GotScraping, OptionsInit, ResponseBodyOnly, TransformHeadersAgent, createGotScrapingOptions, getAgents, gotScraping, hooks };