npm - @node-core/utils - Versions diffs - 7.2.1 → 7.2.2 - Mend

@node-core/utils 7.2.1 → 7.2.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (1354) hide show

package/node_modules/cheerio/node_modules/undici/docs/docs/api/Socks5ProxyAgent.md DELETED Viewed

@@ -1,274 +0,0 @@
-# Class: Socks5ProxyAgent
-Extends: `undici.Dispatcher`
-A SOCKS5 proxy wrapper class that implements the Dispatcher API. It enables HTTP requests to be routed through a SOCKS5 proxy server, providing connection tunneling and authentication support.
-## `new Socks5ProxyAgent(proxyUrl[, options])`
-Arguments:
-* **proxyUrl** `string | URL` (required) - The SOCKS5 proxy server URL. Must use `socks5://` or `socks://` protocol.
-* **options** `Socks5ProxyAgent.Options` (optional) - Additional configuration options.
-Returns: `Socks5ProxyAgent`
-### Parameter: `Socks5ProxyAgent.Options`
-Extends: [`PoolOptions`](/docs/docs/api/Pool.md#parameter-pooloptions)
-* **headers** `IncomingHttpHeaders` (optional) - Additional headers to send with proxy connections.
-* **username** `string` (optional) - SOCKS5 proxy username for authentication. Can also be provided in the proxy URL.
-* **password** `string` (optional) - SOCKS5 proxy password for authentication. Can also be provided in the proxy URL.
-* **connect** `Function` (optional) - Custom connector function for the proxy connection.
-* **proxyTls** `BuildOptions` (optional) - TLS options for the proxy connection (when using SOCKS5 over TLS).
-Examples:
-```js
-import { Socks5ProxyAgent } from 'undici'
-const socks5Proxy = new Socks5ProxyAgent('socks5://localhost:1080')
-// or with authentication
-const socks5ProxyWithAuth = new Socks5ProxyAgent('socks5://user:pass@localhost:1080')
-// or with options
-const socks5ProxyWithOptions = new Socks5ProxyAgent('socks5://localhost:1080', {
-  username: 'user',
-  password: 'pass',
-  connections: 10
-})
-```
-#### Example - Basic SOCKS5 Proxy instantiation
-This will instantiate the Socks5ProxyAgent. It will not do anything until registered as the dispatcher to use with requests.
-```js
-import { Socks5ProxyAgent } from 'undici'
-const socks5Proxy = new Socks5ProxyAgent('socks5://localhost:1080')
-```
-#### Example - Basic SOCKS5 Proxy Request with global dispatcher
-```js
-import { setGlobalDispatcher, request, Socks5ProxyAgent } from 'undici'
-const socks5Proxy = new Socks5ProxyAgent('socks5://localhost:1080')
-setGlobalDispatcher(socks5Proxy)
-const { statusCode, body } = await request('http://localhost:3000/foo')
-console.log('response received', statusCode) // response received 200
-for await (const data of body) {
-  console.log('data', data.toString('utf8')) // data foo
-}
-```
-#### Example - Basic SOCKS5 Proxy Request with local dispatcher
-```js
-import { Socks5ProxyAgent, request } from 'undici'
-const socks5Proxy = new Socks5ProxyAgent('socks5://localhost:1080')
-const {
-  statusCode,
-  body
-} = await request('http://localhost:3000/foo', { dispatcher: socks5Proxy })
-console.log('response received', statusCode) // response received 200
-for await (const data of body) {
-  console.log('data', data.toString('utf8')) // data foo
-}
-```
-#### Example - SOCKS5 Proxy Request with authentication
-```js
-import { setGlobalDispatcher, request, Socks5ProxyAgent } from 'undici'
-// Authentication via URL
-const socks5Proxy = new Socks5ProxyAgent('socks5://username:password@localhost:1080')
-// Or authentication via options
-// const socks5Proxy = new Socks5ProxyAgent('socks5://localhost:1080', {
-//   username: 'username',
-//   password: 'password'
-// })
-setGlobalDispatcher(socks5Proxy)
-const { statusCode, body } = await request('http://localhost:3000/foo')
-console.log('response received', statusCode) // response received 200
-for await (const data of body) {
-  console.log('data', data.toString('utf8')) // data foo
-}
-```
-#### Example - SOCKS5 Proxy with HTTPS requests
-SOCKS5 proxy supports both HTTP and HTTPS requests through tunneling:
-```js
-import { Socks5ProxyAgent, request } from 'undici'
-const socks5Proxy = new Socks5ProxyAgent('socks5://localhost:1080')
-const response = await request('https://api.example.com/data', {
-  dispatcher: socks5Proxy,
-  method: 'GET'
-})
-console.log('Response status:', response.statusCode)
-console.log('Response data:', await response.body.json())
-```
-#### Example - SOCKS5 Proxy with Fetch
-```js
-import { Socks5ProxyAgent, fetch } from 'undici'
-const socks5Proxy = new Socks5ProxyAgent('socks5://localhost:1080')
-const response = await fetch('http://localhost:3000/api/users', {
-  dispatcher: socks5Proxy,
-  method: 'GET'
-})
-console.log('Response status:', response.status)
-console.log('Response data:', await response.text())
-```
-#### Example - Connection Pooling
-SOCKS5ProxyWrapper automatically manages connection pooling for better performance:
-```js
-import { Socks5ProxyAgent, request } from 'undici'
-const socks5Proxy = new Socks5ProxyAgent('socks5://localhost:1080', {
-  connections: 10, // Allow up to 10 concurrent connections
-  pipelining: 1    // Enable HTTP/1.1 pipelining
-})
-// Multiple requests will reuse connections through the SOCKS5 tunnel
-const responses = await Promise.all([
-  request('http://api.example.com/endpoint1', { dispatcher: socks5Proxy }),
-  request('http://api.example.com/endpoint2', { dispatcher: socks5Proxy }),
-  request('http://api.example.com/endpoint3', { dispatcher: socks5Proxy })
-])
-console.log('All requests completed through the same SOCKS5 proxy')
-```
-### `Socks5ProxyAgent.close()`
-Closes the SOCKS5 proxy wrapper and waits for all underlying pools and connections to close before resolving.
-Returns: `Promise<void>`
-#### Example - clean up after tests are complete
-```js
-import { Socks5ProxyAgent, setGlobalDispatcher } from 'undici'
-const socks5Proxy = new Socks5ProxyAgent('socks5://localhost:1080')
-setGlobalDispatcher(socks5Proxy)
-// ... make requests
-await socks5Proxy.close()
-```
-### `Socks5ProxyAgent.destroy([err])`
-Destroys the SOCKS5 proxy wrapper and all underlying connections immediately.
-Arguments:
-* **err** `Error` (optional) - The error that caused the destruction.
-Returns: `Promise<void>`
-#### Example - force close all connections
-```js
-import { Socks5ProxyAgent } from 'undici'
-const socks5Proxy = new Socks5ProxyAgent('socks5://localhost:1080')
-// Force close all connections
-await socks5Proxy.destroy()
-```
-### `Socks5ProxyAgent.dispatch(options, handlers)`
-Implements [`Dispatcher.dispatch(options, handlers)`](/docs/docs/api/Dispatcher.md#dispatcherdispatchoptions-handlers).
-### `Socks5ProxyAgent.request(options[, callback])`
-See [`Dispatcher.request(options [, callback])`](/docs/docs/api/Dispatcher.md#dispatcherrequestoptions-callback).
-## Debugging
-SOCKS5 proxy connections can be debugged using Node.js diagnostics:
-```sh
-NODE_DEBUG=undici:socks5 node script.js
-```
-This will output detailed information about the SOCKS5 handshake, authentication, and connection establishment.
-## SOCKS5 Protocol Support
-The Socks5ProxyAgent supports the following SOCKS5 features:
-### Authentication Methods
-- **No Authentication** (`0x00`) - For public or internal proxies
-- **Username/Password** (`0x02`) - RFC 1929 authentication
-### Address Types
-- **IPv4** (`0x01`) - Standard IPv4 addresses
-- **Domain Name** (`0x03`) - Domain names (recommended for flexibility)
-- **IPv6** (`0x04`) - IPv6 addresses (full support for standard and compressed notation)
-### Commands
-- **CONNECT** (`0x01`) - Establish TCP connection (primary use case for HTTP)
-### Error Handling
-The wrapper handles various SOCKS5 error conditions:
-- Connection refused by proxy
-- Authentication failures
-- Network unreachable
-- Host unreachable
-- Unsupported address types or commands
-## Performance Considerations
-- **Connection Pooling**: Automatically pools connections through the SOCKS5 tunnel for better performance
-- **HTTP/1.1 Pipelining**: Supports pipelining when enabled
-- **DNS Resolution**: Domain names are resolved by the SOCKS5 proxy, reducing local DNS queries
-- **TLS Termination**: HTTPS connections are encrypted end-to-end, with the SOCKS5 proxy only handling the TCP tunnel
-## Security Notes
-1. **Authentication**: Credentials are sent to the SOCKS5 proxy in plaintext unless using SOCKS5 over TLS
-2. **DNS Leaks**: All DNS resolution happens on the proxy server, preventing DNS leaks
-3. **End-to-end Encryption**: HTTPS traffic remains encrypted between client and final destination
-4. **Connection Security**: Consider using authenticated proxies and secure networks
-## Compatibility
-- **Protocol**: SOCKS5 (RFC 1928) with Username/Password Authentication (RFC 1929)
-- **Transport**: TCP only (UDP support not implemented)
-- **Node.js**: Compatible with all supported Node.js versions
-- **HTTP Versions**: Works with HTTP/1.1 and HTTP/2 over the tunnel

package/node_modules/cheerio/node_modules/undici/docs/docs/api/Util.md DELETED Viewed

@@ -1,25 +0,0 @@
-# Util
-Utility API for third-party implementations of the dispatcher API.
-## `parseHeaders(headers, [obj])`
-Receives a header object and returns the parsed value.
-Arguments:
-- **headers** `(Buffer | string | (Buffer | string)[])[]` (required) - Header object.
-- **obj** `Record<string, string | string[]>` (optional) - Object to specify a proxy object. The parsed value is assigned to this object. But, if **headers** is an object, it is not used.
-Returns: `Record<string, string | string[]>` If **obj** is specified, it is equivalent to **obj**.
-## `headerNameToString(value)`
-Retrieves a header name and returns its lowercase value.
-Arguments:
-- **value** `string | Buffer` (required) - Header name.
-Returns: `string`

package/node_modules/cheerio/node_modules/undici/docs/docs/api/WebSocket.md DELETED Viewed

@@ -1,141 +0,0 @@
-# Class: WebSocket
-Extends: [`EventTarget`](https://developer.mozilla.org/en-US/docs/Web/API/EventTarget)
-The WebSocket object provides a way to manage a WebSocket connection to a server, allowing bidirectional communication. The API follows the [WebSocket spec](https://developer.mozilla.org/en-US/docs/Web/API/WebSocket) and [RFC 6455](https://datatracker.ietf.org/doc/html/rfc6455).
-## `new WebSocket(url[, protocol])`
-Arguments:
-* **url** `URL | string`
-* **protocol** `string | string[] | WebSocketInit` (optional) - Subprotocol(s) to request the server use, or a [`Dispatcher`](/docs/docs/api/Dispatcher.md).
-### WebSocketInit
-When passing an object as the second argument, the following options are available:
-* **protocols** `string | string[]` (optional) - Subprotocol(s) to request the server use.
-* **dispatcher** `Dispatcher` (optional) - A custom [`Dispatcher`](/docs/docs/api/Dispatcher.md) to use for the connection.
-* **headers** `HeadersInit` (optional) - Custom headers to include in the WebSocket handshake request.
-### Example:
-This example will not work in browsers or other platforms that don't allow passing an object.
-```js
-import { WebSocket, ProxyAgent } from 'undici'
-const proxyAgent = new ProxyAgent('my.proxy.server')
-const ws = new WebSocket('wss://echo.websocket.events', {
-  dispatcher: proxyAgent,
-  protocols: ['echo', 'chat']
-})
-```
-If you do not need a custom Dispatcher, it's recommended to use the following pattern:
-```js
-import { WebSocket } from 'undici'
-const ws = new WebSocket('wss://echo.websocket.events', ['echo', 'chat'])
-```
-### Example with HTTP/2:
-> ⚠️ Warning: WebSocket over HTTP/2 is experimental, it is likely to change in the future.
-> 🗒️ Note: WebSocket over HTTP/2 may be enabled by default in a future version,
-> this will happen by enabling HTTP/2 connections as the default behavior of Undici's Agent as well the global dispatcher.
-> Stay tuned to the changelog for more information.
-This example will not work in browsers or other platforms that don't allow passing an object.
-```js
-import { Agent } from 'undici'
-const agent = new Agent({ allowH2: true })
-const ws = new WebSocket('wss://echo.websocket.events', {
-  dispatcher: agent,
-  protocols: ['echo', 'chat']
-})
-```
-# Class: WebSocketStream
-> ⚠️ Warning: the WebSocketStream API has not been finalized and is likely to change.
-See [MDN](https://developer.mozilla.org/en-US/docs/Web/API/WebSocketStream) for more information.
-## `new WebSocketStream(url[, protocol])`
-Arguments:
-* **url** `URL | string`
-* **options** `WebSocketStreamOptions` (optional)
-### WebSocketStream Example
-```js
-const stream = new WebSocketStream('https://echo.websocket.org/')
-const { readable, writable } = await stream.opened
-async function read () {
-  /** @type {ReadableStreamReader} */
-  const reader = readable.getReader()
-  while (true) {
-    const { done, value } = await reader.read()
-    if (done) break
-    // do something with value
-  }
-}
-async function write () {
-  /** @type {WritableStreamDefaultWriter} */
-  const writer = writable.getWriter()
-  writer.write('Hello, world!')
-  writer.releaseLock()
-}
-read()
-setInterval(() => write(), 5000)
-```
-## ping(websocket, payload)
-Arguments:
-* **websocket** `WebSocket` - The WebSocket instance to send the ping frame on
-* **payload** `Buffer|undefined` (optional) - Optional payload data to include with the ping frame. Must not exceed 125 bytes.
-Sends a ping frame to the WebSocket server. The server must respond with a pong frame containing the same payload data. This can be used for keepalive purposes or to verify that the connection is still active.
-### Example:
-```js
-import { WebSocket, ping } from 'undici'
-const ws = new WebSocket('wss://echo.websocket.events')
-ws.addEventListener('open', () => {
-  // Send ping with no payload
-  ping(ws)
-  // Send ping with payload
-  const payload = Buffer.from('hello')
-  ping(ws, payload)
-})
-```
-**Note**: A ping frame cannot have a payload larger than 125 bytes. The ping will only be sent if the WebSocket connection is in the OPEN state.
-## Read More
-- [MDN - WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSocket)
-- [The WebSocket Specification](https://www.rfc-editor.org/rfc/rfc6455)
-- [The WHATWG WebSocket Specification](https://websockets.spec.whatwg.org/)

package/node_modules/cheerio/node_modules/undici/docs/docs/api/api-lifecycle.md DELETED Viewed

@@ -1,91 +0,0 @@
-# Client Lifecycle
-An Undici [Client](/docs/docs/api/Client.md) can be best described as a state machine. The following list is a summary of the various state transitions the `Client` will go through in its lifecycle. This document also contains detailed breakdowns of each state.
-> This diagram is not a perfect representation of the undici Client. Since the Client class is not actually implemented as a state-machine, actual execution may deviate slightly from what is described below. Consider this as a general resource for understanding the inner workings of the Undici client rather than some kind of formal specification.
-## State Transition Overview
-* A `Client` begins in the **idle** state with no socket connection and no requests in queue.
-  * The *connect* event transitions the `Client` to the **pending** state where requests can be queued prior to processing.
-  * The *close* and *destroy* events transition the `Client` to the **destroyed** state. Since there are no requests in the queue, the *close* event immediately transitions to the **destroyed** state.
-* The **pending** state indicates the underlying socket connection has been successfully established and requests are queueing.
-  * The *process* event transitions the `Client` to the **processing** state where requests are processed.
-  * If requests are queued, the *close* event transitions to the **processing** state; otherwise, it transitions to the **destroyed** state.
-  * The *destroy* event transitions to the **destroyed** state.
-* The **processing** state initializes to the **processing.running** state.
-  * If the current request requires draining, the *needDrain* event transitions the `Client` into the **processing.busy** state which will return to the **processing.running** state with the *drainComplete* event.
-  * After all queued requests are completed, the *keepalive* event transitions the `Client` back to the **pending** state. If no requests are queued during the timeout, the **close** event transitions the `Client` to the **destroyed** state.
-  * If the *close* event is fired while the `Client` still has queued requests, the `Client` transitions to the **process.closing** state where it will complete all existing requests before firing the *done* event.
-  * The *done* event gracefully transitions the `Client` to the **destroyed** state.
-  * At any point in time, the *destroy* event will transition the `Client` from the **processing** state to the **destroyed** state, destroying any queued requests.
-* The **destroyed** state is a final state and the `Client` is no longer functional.
-A state diagram representing an Undici Client instance:
-```mermaid
-stateDiagram-v2
-  [*] --> idle
-  idle --> pending : connect
-  idle --> destroyed : destroy/close
-  pending --> idle : timeout
-  pending --> destroyed : destroy
-  state close_fork <<fork>>
-  pending --> close_fork : close
-  close_fork --> processing
-  close_fork --> destroyed
-  pending --> processing : process
-  processing --> pending : keepalive
-  processing --> destroyed : done
-  processing --> destroyed : destroy
-  destroyed --> [*]
-  state processing {
-      [*] --> running
-      running --> closing : close
-      running --> busy : needDrain
-      busy --> running : drainComplete
-      running --> [*] : keepalive
-      closing --> [*] : done
-  }
-```
-## State details
-### idle
-The **idle** state is the initial state of a `Client` instance. While an `origin` is required for instantiating a `Client` instance, the underlying socket connection will not be established until a request is queued using [`Client.dispatch()`](/docs/docs/api/Client.md#clientdispatchoptions-handlers). By calling `Client.dispatch()` directly or using one of the multiple implementations ([`Client.connect()`](Client.md#clientconnectoptions-callback), [`Client.pipeline()`](Client.md#clientpipelineoptions-handler), [`Client.request()`](Client.md#clientrequestoptions-callback), [`Client.stream()`](Client.md#clientstreamoptions-factory-callback), and [`Client.upgrade()`](/docs/docs/api/Client.md#clientupgradeoptions-callback)), the `Client` instance will transition from **idle** to [**pending**](/docs/docs/api/Client.md#pending) and then most likely directly to [**processing**](/docs/docs/api/Client.md#processing).
-Calling [`Client.close()`](/docs/docs/api/Client.md#clientclosecallback) or [`Client.destroy()`](Client.md#clientdestroyerror-callback) transitions directly to the [**destroyed**](/docs/docs/api/Client.md#destroyed) state since the `Client` instance will have no queued requests in this state.
-### pending
-The **pending** state signifies a non-processing `Client`. Upon entering this state, the `Client` establishes a socket connection and emits the [`'connect'`](/docs/docs/api/Client.md#event-connect) event signalling a connection was successfully established with the `origin` provided during `Client` instantiation. The internal queue is initially empty, and requests can start queueing.
-Calling [`Client.close()`](/docs/docs/api/Client.md#clientclosecallback) with queued requests, transitions the `Client` to the [**processing**](/docs/docs/api/Client.md#processing) state. Without queued requests, it transitions to the [**destroyed**](/docs/docs/api/Client.md#destroyed) state.
-Calling [`Client.destroy()`](/docs/docs/api/Client.md#clientdestroyerror-callback) transitions directly to the [**destroyed**](/docs/docs/api/Client.md#destroyed) state regardless of existing requests.
-### processing
-The **processing** state is a state machine within itself. It initializes to the [**processing.running**](/docs/docs/api/Client.md#running) state. The [`Client.dispatch()`](/docs/docs/api/Client.md#clientdispatchoptions-handlers), [`Client.close()`](Client.md#clientclosecallback), and [`Client.destroy()`](Client.md#clientdestroyerror-callback) can be called at any time while the `Client` is in this state. `Client.dispatch()` will add more requests to the queue while existing requests continue to be processed. `Client.close()` will transition to the [**processing.closing**](/docs/docs/api/Client.md#closing) state. And `Client.destroy()` will transition to [**destroyed**](/docs/docs/api/Client.md#destroyed).
-#### running
-In the **processing.running** sub-state, queued requests are being processed in a FIFO order. If a request body requires draining, the *needDrain* event transitions to the [**processing.busy**](/docs/docs/api/Client.md#busy) sub-state. The *close* event transitions the Client to the [**process.closing**](/docs/docs/api/Client.md#closing) sub-state. If all queued requests are processed and neither [`Client.close()`](/docs/docs/api/Client.md#clientclosecallback) nor [`Client.destroy()`](Client.md#clientdestroyerror-callback) are called, then the [**processing**](/docs/docs/api/Client.md#processing) machine will trigger a *keepalive* event transitioning the `Client` back to the [**pending**](/docs/docs/api/Client.md#pending) state. During this time, the `Client` is waiting for the socket connection to timeout, and once it does, it triggers the *timeout* event and transitions to the [**idle**](/docs/docs/api/Client.md#idle) state.
-#### busy
-This sub-state is only entered when a request body is an instance of [Stream](https://nodejs.org/api/stream.html) and requires draining. The `Client` cannot process additional requests while in this state and must wait until the currently processing request body is completely drained before transitioning back to [**processing.running**](/docs/docs/api/Client.md#running).
-#### closing
-This sub-state is only entered when a `Client` instance has queued requests and the [`Client.close()`](/docs/docs/api/Client.md#clientclosecallback) method is called. In this state, the `Client` instance continues to process requests as usual, with the one exception that no additional requests can be queued. Once all of the queued requests are processed, the `Client` will trigger the *done* event gracefully entering the [**destroyed**](/docs/docs/api/Client.md#destroyed) state without an error.
-### destroyed
-The **destroyed** state is a final state for the `Client` instance. Once in this state, a `Client` is nonfunctional. Calling any other `Client` methods will result in an `ClientDestroyedError`.

package/node_modules/cheerio/node_modules/undici/docs/docs/best-practices/client-certificate.md DELETED Viewed

@@ -1,64 +0,0 @@
-# Client certificate
-Client certificate authentication can be configured with the `Client`, the required options are passed along through the `connect` option.
-The client certificates must be signed by a trusted CA. The Node.js default is to trust the well-known CAs curated by Mozilla.
-Setting the server option `requestCert: true` tells the server to request the client certificate.
-The server option `rejectUnauthorized: false` allows us to handle any invalid certificate errors in client code. The `authorized` property on the socket of the incoming request will show if the client certificate was valid. The `authorizationError` property will give the reason if the certificate was not valid.
-### Client Certificate Authentication
-```js
-const { readFileSync } = require('node:fs')
-const { join } = require('node:path')
-const { createServer } = require('node:https')
-const { Client } = require('undici')
-const serverOptions = {
-  ca: [
-    readFileSync(join(__dirname, 'client-ca-crt.pem'), 'utf8')
-  ],
-  key: readFileSync(join(__dirname, 'server-key.pem'), 'utf8'),
-  cert: readFileSync(join(__dirname, 'server-crt.pem'), 'utf8'),
-  requestCert: true,
-  rejectUnauthorized: false
-}
-const server = createServer(serverOptions, (req, res) => {
-  // true if client cert is valid
-  if(req.client.authorized === true) {
-    console.log('valid')
-  } else {
-    console.error(req.client.authorizationError)
-  }
-  res.end()
-})
-server.listen(0, function () {
-  const tls = {
-    ca: [
-      readFileSync(join(__dirname, 'server-ca-crt.pem'), 'utf8')
-    ],
-    key: readFileSync(join(__dirname, 'client-key.pem'), 'utf8'),
-    cert: readFileSync(join(__dirname, 'client-crt.pem'), 'utf8'),
-    rejectUnauthorized: false,
-    servername: 'agent1'
-  }
-  const client = new Client(`https://localhost:${server.address().port}`, {
-    connect: tls
-  })
-  client.request({
-    path: '/',
-    method: 'GET'
-  }, (err, { body }) => {
-    body.on('data', (buf) => {})
-    body.on('end', () => {
-      client.close()
-      server.close()
-    })
-  })
-})
-```

package/node_modules/cheerio/node_modules/undici/docs/docs/best-practices/crawling.md DELETED Viewed

@@ -1,58 +0,0 @@
-# Crawling
-[RFC 9309](https://datatracker.ietf.org/doc/html/rfc9309) defines crawlers as automated clients.
-Some web servers may reject requests that omit the `User-Agent` header or that use common defaults such as `'curl/7.79.1'`.
-In **undici**, the default user agent is `'undici'`. Since undici is integrated into Node.js core as the implementation of `fetch()`, requests made via `fetch()` use `'node'` as the default user agent.
-It is recommended to specify a **custom `User-Agent` header** when implementing crawlers. Providing a descriptive user agent allows servers to correctly identify the client and reduces the likelihood of requests being denied.
-A user agent string should include sufficient detail to identify the crawler and provide contact information. For example:
-```
-AcmeCo Crawler - acme.co - contact@acme.co
-```
-When adding contact details, avoid using personal identifiers such as your own name or a private email address—especially in a professional or employment context. Instead, use a role-based or organizational contact (e.g., crawler-team@company.com) to protect individual privacy while still enabling communication.
-If a crawler behaves unexpectedly—for example, due to misconfiguration or implementation errors—server administrators can use the information in the user agent to contact the operator and coordinate an appropriate resolution.
-The `User-Agent` header can be set on individual requests or applied globally by configuring a custom dispatcher.
-**Example: setting a `User-Agent` per request**
-```js
-import { fetch } from 'undici'
-const headers = {
-  'User-Agent': 'AcmeCo Crawler - acme.co - contact@acme.co'
-}
-const res = await fetch('https://example.com', { headers })
-```
-## Best Practices for Crawlers
-When developing a crawler, the following practices are recommended in addition to setting a descriptive `User-Agent` header:
-* **Respect `robots.txt`**
-  Follow the directives defined in the target site’s `robots.txt` file, including disallowed paths and optional crawl-delay settings (see [W3C guidelines](https://www.w3.org/wiki/Write_Web_Crawler)).
-* **Rate limiting**
-  Regulate request frequency to avoid imposing excessive load on servers. Introduce delays between requests or limit the number of concurrent requests. The W3C suggests at least one second between requests.
-* **Error handling**
-  Implement retry logic with exponential backoff for transient failures, and stop requests when persistent errors occur (e.g., HTTP 403 or 429).
-* **Monitoring and logging**
-  Track request volume, response codes, and error rates to detect misbehavior and address issues proactively.
-* **Contact information**
-  Always include valid and current contact details in the `User-Agent` string so that administrators can reach the crawler operator if necessary.
-## References and Further Reading
-* [RFC 9309: The Robots Exclusion Protocol](https://datatracker.ietf.org/doc/html/rfc9309)
-* [W3C Wiki: Write Web Crawler](https://www.w3.org/wiki/Write_Web_Crawler)
-* [Ethical Web Crawling (WWW 2010 Conference Paper)](https://archives.iw3c2.org/www2010/proceedings/www/p1101.pdf)