@dragon708/docmind-markdown 1.2.0 → 1.2.2

package/node_modules/undici/docs/docs/api/Socks5ProxyAgent.md

@@ -0,0 +1,274 @@
+# Class: Socks5ProxyAgent
+
+Extends: `undici.Dispatcher`
+
+A SOCKS5 proxy wrapper class that implements the Dispatcher API. It enables HTTP requests to be routed through a SOCKS5 proxy server, providing connection tunneling and authentication support.
+
+## `new Socks5ProxyAgent(proxyUrl[, options])`
+
+Arguments:
+
+* **proxyUrl** `string | URL` (required) - The SOCKS5 proxy server URL. Must use `socks5://` or `socks://` protocol.
+* **options** `Socks5ProxyAgent.Options` (optional) - Additional configuration options.
+
+Returns: `Socks5ProxyAgent`
+
+### Parameter: `Socks5ProxyAgent.Options`
+
+Extends: [`PoolOptions`](/docs/docs/api/Pool.md#parameter-pooloptions)
+
+* **headers** `IncomingHttpHeaders` (optional) - Additional headers to send with proxy connections.
+* **username** `string` (optional) - SOCKS5 proxy username for authentication. Can also be provided in the proxy URL.
+* **password** `string` (optional) - SOCKS5 proxy password for authentication. Can also be provided in the proxy URL.
+* **connect** `Function` (optional) - Custom connector function for the proxy connection.
+* **proxyTls** `BuildOptions` (optional) - TLS options for the proxy connection (when using SOCKS5 over TLS).
+
+Examples:
+
+```js
+import { Socks5ProxyAgent } from 'undici'
+
+const socks5Proxy = new Socks5ProxyAgent('socks5://localhost:1080')
+// or with authentication
+const socks5ProxyWithAuth = new Socks5ProxyAgent('socks5://user:pass@localhost:1080')
+// or with options
+const socks5ProxyWithOptions = new Socks5ProxyAgent('socks5://localhost:1080', {
+  username: 'user',
+  password: 'pass',
+  connections: 10
+})
+```
+
+#### Example - Basic SOCKS5 Proxy instantiation
+
+This will instantiate the Socks5ProxyAgent. It will not do anything until registered as the dispatcher to use with requests.
+
+```js
+import { Socks5ProxyAgent } from 'undici'
+
+const socks5Proxy = new Socks5ProxyAgent('socks5://localhost:1080')
+```
+
+#### Example - Basic SOCKS5 Proxy Request with global dispatcher
+
+```js
+import { setGlobalDispatcher, request, Socks5ProxyAgent } from 'undici'
+
+const socks5Proxy = new Socks5ProxyAgent('socks5://localhost:1080')
+setGlobalDispatcher(socks5Proxy)
+
+const { statusCode, body } = await request('http://localhost:3000/foo')
+
+console.log('response received', statusCode) // response received 200
+
+for await (const data of body) {
+  console.log('data', data.toString('utf8')) // data foo
+}
+```
+
+#### Example - Basic SOCKS5 Proxy Request with local dispatcher
+
+```js
+import { Socks5ProxyAgent, request } from 'undici'
+
+const socks5Proxy = new Socks5ProxyAgent('socks5://localhost:1080')
+
+const {
+  statusCode,
+  body
+} = await request('http://localhost:3000/foo', { dispatcher: socks5Proxy })
+
+console.log('response received', statusCode) // response received 200
+
+for await (const data of body) {
+  console.log('data', data.toString('utf8')) // data foo
+}
+```
+
+#### Example - SOCKS5 Proxy Request with authentication
+
+```js
+import { setGlobalDispatcher, request, Socks5ProxyAgent } from 'undici'
+
+// Authentication via URL
+const socks5Proxy = new Socks5ProxyAgent('socks5://username:password@localhost:1080')
+
+// Or authentication via options
+// const socks5Proxy = new Socks5ProxyAgent('socks5://localhost:1080', {
+//   username: 'username',
+//   password: 'password'
+// })
+
+setGlobalDispatcher(socks5Proxy)
+
+const { statusCode, body } = await request('http://localhost:3000/foo')
+
+console.log('response received', statusCode) // response received 200
+
+for await (const data of body) {
+  console.log('data', data.toString('utf8')) // data foo
+}
+```
+
+#### Example - SOCKS5 Proxy with HTTPS requests
+
+SOCKS5 proxy supports both HTTP and HTTPS requests through tunneling:
+
+```js
+import { Socks5ProxyAgent, request } from 'undici'
+
+const socks5Proxy = new Socks5ProxyAgent('socks5://localhost:1080')
+
+const response = await request('https://api.example.com/data', {
+  dispatcher: socks5Proxy,
+  method: 'GET'
+})
+
+console.log('Response status:', response.statusCode)
+console.log('Response data:', await response.body.json())
+```
+
+#### Example - SOCKS5 Proxy with Fetch
+
+```js
+import { Socks5ProxyAgent, fetch } from 'undici'
+
+const socks5Proxy = new Socks5ProxyAgent('socks5://localhost:1080')
+
+const response = await fetch('http://localhost:3000/api/users', {
+  dispatcher: socks5Proxy,
+  method: 'GET'
+})
+
+console.log('Response status:', response.status)
+console.log('Response data:', await response.text())
+```
+
+#### Example - Connection Pooling
+
+SOCKS5ProxyWrapper automatically manages connection pooling for better performance:
+
+```js
+import { Socks5ProxyAgent, request } from 'undici'
+
+const socks5Proxy = new Socks5ProxyAgent('socks5://localhost:1080', {
+  connections: 10, // Allow up to 10 concurrent connections
+  pipelining: 1    // Enable HTTP/1.1 pipelining
+})
+
+// Multiple requests will reuse connections through the SOCKS5 tunnel
+const responses = await Promise.all([
+  request('http://api.example.com/endpoint1', { dispatcher: socks5Proxy }),
+  request('http://api.example.com/endpoint2', { dispatcher: socks5Proxy }),
+  request('http://api.example.com/endpoint3', { dispatcher: socks5Proxy })
+])
+
+console.log('All requests completed through the same SOCKS5 proxy')
+```
+
+### `Socks5ProxyAgent.close()`
+
+Closes the SOCKS5 proxy wrapper and waits for all underlying pools and connections to close before resolving.
+
+Returns: `Promise<void>`
+
+#### Example - clean up after tests are complete
+
+```js
+import { Socks5ProxyAgent, setGlobalDispatcher } from 'undici'
+
+const socks5Proxy = new Socks5ProxyAgent('socks5://localhost:1080')
+setGlobalDispatcher(socks5Proxy)
+
+// ... make requests
+
+await socks5Proxy.close()
+```
+
+### `Socks5ProxyAgent.destroy([err])`
+
+Destroys the SOCKS5 proxy wrapper and all underlying connections immediately.
+
+Arguments:
+* **err** `Error` (optional) - The error that caused the destruction.
+
+Returns: `Promise<void>`
+
+#### Example - force close all connections
+
+```js
+import { Socks5ProxyAgent } from 'undici'
+
+const socks5Proxy = new Socks5ProxyAgent('socks5://localhost:1080')
+
+// Force close all connections
+await socks5Proxy.destroy()
+```
+
+### `Socks5ProxyAgent.dispatch(options, handlers)`
+
+Implements [`Dispatcher.dispatch(options, handlers)`](/docs/docs/api/Dispatcher.md#dispatcherdispatchoptions-handlers).
+
+### `Socks5ProxyAgent.request(options[, callback])`
+
+See [`Dispatcher.request(options [, callback])`](/docs/docs/api/Dispatcher.md#dispatcherrequestoptions-callback).
+
+## Debugging
+
+SOCKS5 proxy connections can be debugged using Node.js diagnostics:
+
+```sh
+NODE_DEBUG=undici:socks5 node script.js
+```
+
+This will output detailed information about the SOCKS5 handshake, authentication, and connection establishment.
+
+## SOCKS5 Protocol Support
+
+The Socks5ProxyAgent supports the following SOCKS5 features:
+
+### Authentication Methods
+
+- **No Authentication** (`0x00`) - For public or internal proxies
+- **Username/Password** (`0x02`) - RFC 1929 authentication
+
+### Address Types
+
+- **IPv4** (`0x01`) - Standard IPv4 addresses
+- **Domain Name** (`0x03`) - Domain names (recommended for flexibility)
+- **IPv6** (`0x04`) - IPv6 addresses (full support for standard and compressed notation)
+
+### Commands
+
+- **CONNECT** (`0x01`) - Establish TCP connection (primary use case for HTTP)
+
+### Error Handling
+
+The wrapper handles various SOCKS5 error conditions:
+
+- Connection refused by proxy
+- Authentication failures
+- Network unreachable
+- Host unreachable
+- Unsupported address types or commands
+
+## Performance Considerations
+
+- **Connection Pooling**: Automatically pools connections through the SOCKS5 tunnel for better performance
+- **HTTP/1.1 Pipelining**: Supports pipelining when enabled
+- **DNS Resolution**: Domain names are resolved by the SOCKS5 proxy, reducing local DNS queries
+- **TLS Termination**: HTTPS connections are encrypted end-to-end, with the SOCKS5 proxy only handling the TCP tunnel
+
+## Security Notes
+
+1. **Authentication**: Credentials are sent to the SOCKS5 proxy in plaintext unless using SOCKS5 over TLS
+2. **DNS Leaks**: All DNS resolution happens on the proxy server, preventing DNS leaks
+3. **End-to-end Encryption**: HTTPS traffic remains encrypted between client and final destination
+4. **Connection Security**: Consider using authenticated proxies and secure networks
+
+## Compatibility
+
+- **Protocol**: SOCKS5 (RFC 1928) with Username/Password Authentication (RFC 1929)
+- **Transport**: TCP only (UDP support not implemented)
+- **Node.js**: Compatible with all supported Node.js versions
+- **HTTP Versions**: Works with HTTP/1.1 and HTTP/2 over the tunnel

package/node_modules/undici/docs/docs/api/Util.md

@@ -0,0 +1,25 @@
+# Util
+
+Utility API for third-party implementations of the dispatcher API.
+
+## `parseHeaders(headers, [obj])`
+
+Receives a header object and returns the parsed value.
+
+Arguments:
+
+- **headers** `(Buffer | string | (Buffer | string)[])[]` (required) - Header object.
+
+- **obj** `Record<string, string | string[]>` (optional) - Object to specify a proxy object. The parsed value is assigned to this object. But, if **headers** is an object, it is not used.
+
+Returns: `Record<string, string | string[]>` If **obj** is specified, it is equivalent to **obj**.
+
+## `headerNameToString(value)`
+
+Retrieves a header name and returns its lowercase value.
+
+Arguments:
+
+- **value** `string | Buffer` (required) - Header name.
+
+Returns: `string`

package/node_modules/undici/docs/docs/api/WebSocket.md

@@ -0,0 +1,141 @@
+# Class: WebSocket
+
+Extends: [`EventTarget`](https://developer.mozilla.org/en-US/docs/Web/API/EventTarget)
+
+The WebSocket object provides a way to manage a WebSocket connection to a server, allowing bidirectional communication. The API follows the [WebSocket spec](https://developer.mozilla.org/en-US/docs/Web/API/WebSocket) and [RFC 6455](https://datatracker.ietf.org/doc/html/rfc6455).
+
+## `new WebSocket(url[, protocol])`
+
+Arguments:
+
+* **url** `URL | string`
+* **protocol** `string | string[] | WebSocketInit` (optional) - Subprotocol(s) to request the server use, or a [`Dispatcher`](/docs/docs/api/Dispatcher.md).
+
+### WebSocketInit
+
+When passing an object as the second argument, the following options are available:
+
+* **protocols** `string | string[]` (optional) - Subprotocol(s) to request the server use.
+* **dispatcher** `Dispatcher` (optional) - A custom [`Dispatcher`](/docs/docs/api/Dispatcher.md) to use for the connection.
+* **headers** `HeadersInit` (optional) - Custom headers to include in the WebSocket handshake request.
+
+### Example:
+
+This example will not work in browsers or other platforms that don't allow passing an object.
+
+```js
+import { WebSocket, ProxyAgent } from 'undici'
+
+const proxyAgent = new ProxyAgent('my.proxy.server')
+
+const ws = new WebSocket('wss://echo.websocket.events', {
+  dispatcher: proxyAgent,
+  protocols: ['echo', 'chat']
+})
+```
+
+If you do not need a custom Dispatcher, it's recommended to use the following pattern:
+
+```js
+import { WebSocket } from 'undici'
+
+const ws = new WebSocket('wss://echo.websocket.events', ['echo', 'chat'])
+```
+
+### Example with HTTP/2:
+
+> ⚠️ Warning: WebSocket over HTTP/2 is experimental, it is likely to change in the future.
+
+> 🗒️ Note: WebSocket over HTTP/2 may be enabled by default in a future version,
+> this will happen by enabling HTTP/2 connections as the default behavior of Undici's Agent as well the global dispatcher.
+> Stay tuned to the changelog for more information.
+
+This example will not work in browsers or other platforms that don't allow passing an object.
+
+```js
+import { Agent } from 'undici'
+
+const agent = new Agent({ allowH2: true })
+
+const ws = new WebSocket('wss://echo.websocket.events', {
+  dispatcher: agent,
+  protocols: ['echo', 'chat']
+})
+```
+
+# Class: WebSocketStream
+
+> ⚠️ Warning: the WebSocketStream API has not been finalized and is likely to change.
+
+See [MDN](https://developer.mozilla.org/en-US/docs/Web/API/WebSocketStream) for more information.
+
+## `new WebSocketStream(url[, protocol])`
+
+Arguments:
+
+* **url** `URL | string`
+* **options** `WebSocketStreamOptions` (optional)
+
+### WebSocketStream Example
+
+```js
+const stream = new WebSocketStream('https://echo.websocket.org/')
+const { readable, writable } = await stream.opened
+
+async function read () {
+  /** @type {ReadableStreamReader} */
+  const reader = readable.getReader()
+
+  while (true) {
+    const { done, value } = await reader.read()
+    if (done) break
+
+    // do something with value
+  }
+}
+
+async function write () {
+  /** @type {WritableStreamDefaultWriter} */
+  const writer = writable.getWriter()
+  writer.write('Hello, world!')
+  writer.releaseLock()
+}
+
+read()
+
+setInterval(() => write(), 5000)
+
+```
+
+## ping(websocket, payload)
+Arguments:
+
+* **websocket** `WebSocket` - The WebSocket instance to send the ping frame on
+* **payload** `Buffer|undefined` (optional) - Optional payload data to include with the ping frame. Must not exceed 125 bytes.
+
+Sends a ping frame to the WebSocket server. The server must respond with a pong frame containing the same payload data. This can be used for keepalive purposes or to verify that the connection is still active.
+
+### Example:
+
+```js
+import { WebSocket, ping } from 'undici'
+
+const ws = new WebSocket('wss://echo.websocket.events')
+
+ws.addEventListener('open', () => {
+  // Send ping with no payload
+  ping(ws)
+
+  // Send ping with payload
+  const payload = Buffer.from('hello')
+  ping(ws, payload)
+})
+```
+
+**Note**: A ping frame cannot have a payload larger than 125 bytes. The ping will only be sent if the WebSocket connection is in the OPEN state.
+
+## Read More
+
+- [MDN - WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSocket)
+- [The WebSocket Specification](https://www.rfc-editor.org/rfc/rfc6455)
+- [The WHATWG WebSocket Specification](https://websockets.spec.whatwg.org/)

package/node_modules/undici/docs/docs/api/api-lifecycle.md

@@ -0,0 +1,91 @@
+# Client Lifecycle
+
+An Undici [Client](/docs/docs/api/Client.md) can be best described as a state machine. The following list is a summary of the various state transitions the `Client` will go through in its lifecycle. This document also contains detailed breakdowns of each state.
+
+> This diagram is not a perfect representation of the undici Client. Since the Client class is not actually implemented as a state-machine, actual execution may deviate slightly from what is described below. Consider this as a general resource for understanding the inner workings of the Undici client rather than some kind of formal specification.
+
+## State Transition Overview
+
+* A `Client` begins in the **idle** state with no socket connection and no requests in queue.
+  * The *connect* event transitions the `Client` to the **pending** state where requests can be queued prior to processing.
+  * The *close* and *destroy* events transition the `Client` to the **destroyed** state. Since there are no requests in the queue, the *close* event immediately transitions to the **destroyed** state.
+* The **pending** state indicates the underlying socket connection has been successfully established and requests are queueing.
+  * The *process* event transitions the `Client` to the **processing** state where requests are processed.
+  * If requests are queued, the *close* event transitions to the **processing** state; otherwise, it transitions to the **destroyed** state.
+  * The *destroy* event transitions to the **destroyed** state.
+* The **processing** state initializes to the **processing.running** state.
+  * If the current request requires draining, the *needDrain* event transitions the `Client` into the **processing.busy** state which will return to the **processing.running** state with the *drainComplete* event.
+  * After all queued requests are completed, the *keepalive* event transitions the `Client` back to the **pending** state. If no requests are queued during the timeout, the **close** event transitions the `Client` to the **destroyed** state.
+  * If the *close* event is fired while the `Client` still has queued requests, the `Client` transitions to the **process.closing** state where it will complete all existing requests before firing the *done* event.
+  * The *done* event gracefully transitions the `Client` to the **destroyed** state.
+  * At any point in time, the *destroy* event will transition the `Client` from the **processing** state to the **destroyed** state, destroying any queued requests.
+* The **destroyed** state is a final state and the `Client` is no longer functional.
+
+A state diagram representing an Undici Client instance:
+
+```mermaid
+stateDiagram-v2
+  [*] --> idle
+  idle --> pending : connect
+  idle --> destroyed : destroy/close
+
+  pending --> idle : timeout
+  pending --> destroyed : destroy
+
+  state close_fork <<fork>>
+  pending --> close_fork : close
+  close_fork --> processing
+  close_fork --> destroyed
+
+  pending --> processing : process
+
+  processing --> pending : keepalive
+  processing --> destroyed : done
+  processing --> destroyed : destroy
+
+  destroyed --> [*]
+
+  state processing {
+      [*] --> running
+      running --> closing : close
+      running --> busy : needDrain
+      busy --> running : drainComplete
+      running --> [*] : keepalive
+      closing --> [*] : done
+  }
+```
+## State details
+
+### idle
+
+The **idle** state is the initial state of a `Client` instance. While an `origin` is required for instantiating a `Client` instance, the underlying socket connection will not be established until a request is queued using [`Client.dispatch()`](/docs/docs/api/Client.md#clientdispatchoptions-handlers). By calling `Client.dispatch()` directly or using one of the multiple implementations ([`Client.connect()`](Client.md#clientconnectoptions-callback), [`Client.pipeline()`](Client.md#clientpipelineoptions-handler), [`Client.request()`](Client.md#clientrequestoptions-callback), [`Client.stream()`](Client.md#clientstreamoptions-factory-callback), and [`Client.upgrade()`](/docs/docs/api/Client.md#clientupgradeoptions-callback)), the `Client` instance will transition from **idle** to [**pending**](/docs/docs/api/Client.md#pending) and then most likely directly to [**processing**](/docs/docs/api/Client.md#processing).
+
+Calling [`Client.close()`](/docs/docs/api/Client.md#clientclosecallback) or [`Client.destroy()`](Client.md#clientdestroyerror-callback) transitions directly to the [**destroyed**](/docs/docs/api/Client.md#destroyed) state since the `Client` instance will have no queued requests in this state.
+
+### pending
+
+The **pending** state signifies a non-processing `Client`. Upon entering this state, the `Client` establishes a socket connection and emits the [`'connect'`](/docs/docs/api/Client.md#event-connect) event signalling a connection was successfully established with the `origin` provided during `Client` instantiation. The internal queue is initially empty, and requests can start queueing.
+
+Calling [`Client.close()`](/docs/docs/api/Client.md#clientclosecallback) with queued requests, transitions the `Client` to the [**processing**](/docs/docs/api/Client.md#processing) state. Without queued requests, it transitions to the [**destroyed**](/docs/docs/api/Client.md#destroyed) state.
+
+Calling [`Client.destroy()`](/docs/docs/api/Client.md#clientdestroyerror-callback) transitions directly to the [**destroyed**](/docs/docs/api/Client.md#destroyed) state regardless of existing requests.
+
+### processing
+
+The **processing** state is a state machine within itself. It initializes to the [**processing.running**](/docs/docs/api/Client.md#running) state. The [`Client.dispatch()`](/docs/docs/api/Client.md#clientdispatchoptions-handlers), [`Client.close()`](Client.md#clientclosecallback), and [`Client.destroy()`](Client.md#clientdestroyerror-callback) can be called at any time while the `Client` is in this state. `Client.dispatch()` will add more requests to the queue while existing requests continue to be processed. `Client.close()` will transition to the [**processing.closing**](/docs/docs/api/Client.md#closing) state. And `Client.destroy()` will transition to [**destroyed**](/docs/docs/api/Client.md#destroyed).
+
+#### running
+
+In the **processing.running** sub-state, queued requests are being processed in a FIFO order. If a request body requires draining, the *needDrain* event transitions to the [**processing.busy**](/docs/docs/api/Client.md#busy) sub-state. The *close* event transitions the Client to the [**process.closing**](/docs/docs/api/Client.md#closing) sub-state. If all queued requests are processed and neither [`Client.close()`](/docs/docs/api/Client.md#clientclosecallback) nor [`Client.destroy()`](Client.md#clientdestroyerror-callback) are called, then the [**processing**](/docs/docs/api/Client.md#processing) machine will trigger a *keepalive* event transitioning the `Client` back to the [**pending**](/docs/docs/api/Client.md#pending) state. During this time, the `Client` is waiting for the socket connection to timeout, and once it does, it triggers the *timeout* event and transitions to the [**idle**](/docs/docs/api/Client.md#idle) state.
+
+#### busy
+
+This sub-state is only entered when a request body is an instance of [Stream](https://nodejs.org/api/stream.html) and requires draining. The `Client` cannot process additional requests while in this state and must wait until the currently processing request body is completely drained before transitioning back to [**processing.running**](/docs/docs/api/Client.md#running).
+
+#### closing
+
+This sub-state is only entered when a `Client` instance has queued requests and the [`Client.close()`](/docs/docs/api/Client.md#clientclosecallback) method is called. In this state, the `Client` instance continues to process requests as usual, with the one exception that no additional requests can be queued. Once all of the queued requests are processed, the `Client` will trigger the *done* event gracefully entering the [**destroyed**](/docs/docs/api/Client.md#destroyed) state without an error.
+
+### destroyed
+
+The **destroyed** state is a final state for the `Client` instance. Once in this state, a `Client` is nonfunctional. Calling any other `Client` methods will result in an `ClientDestroyedError`.

package/node_modules/undici/docs/docs/best-practices/client-certificate.md

@@ -0,0 +1,64 @@
+# Client certificate
+
+Client certificate authentication can be configured with the `Client`, the required options are passed along through the `connect` option.
+
+The client certificates must be signed by a trusted CA. The Node.js default is to trust the well-known CAs curated by Mozilla.
+
+Setting the server option `requestCert: true` tells the server to request the client certificate.
+
+The server option `rejectUnauthorized: false` allows us to handle any invalid certificate errors in client code. The `authorized` property on the socket of the incoming request will show if the client certificate was valid. The `authorizationError` property will give the reason if the certificate was not valid.
+
+### Client Certificate Authentication
+
+```js
+const { readFileSync } = require('node:fs')
+const { join } = require('node:path')
+const { createServer } = require('node:https')
+const { Client } = require('undici')
+
+const serverOptions = {
+  ca: [
+    readFileSync(join(__dirname, 'client-ca-crt.pem'), 'utf8')
+  ],
+  key: readFileSync(join(__dirname, 'server-key.pem'), 'utf8'),
+  cert: readFileSync(join(__dirname, 'server-crt.pem'), 'utf8'),
+  requestCert: true,
+  rejectUnauthorized: false
+}
+
+const server = createServer(serverOptions, (req, res) => {
+  // true if client cert is valid
+  if(req.client.authorized === true) {
+    console.log('valid')
+  } else {
+    console.error(req.client.authorizationError)
+  }
+  res.end()
+})
+
+server.listen(0, function () {
+  const tls = {
+    ca: [
+      readFileSync(join(__dirname, 'server-ca-crt.pem'), 'utf8')
+    ],
+    key: readFileSync(join(__dirname, 'client-key.pem'), 'utf8'),
+    cert: readFileSync(join(__dirname, 'client-crt.pem'), 'utf8'),
+    rejectUnauthorized: false,
+    servername: 'agent1'
+  }
+  const client = new Client(`https://localhost:${server.address().port}`, {
+    connect: tls
+  })
+
+  client.request({
+    path: '/',
+    method: 'GET'
+  }, (err, { body }) => {
+    body.on('data', (buf) => {})
+    body.on('end', () => {
+      client.close()
+      server.close()
+    })
+  })
+})
+```

package/node_modules/undici/docs/docs/best-practices/crawling.md

@@ -0,0 +1,58 @@
+# Crawling
+
+[RFC 9309](https://datatracker.ietf.org/doc/html/rfc9309) defines crawlers as automated clients.
+
+Some web servers may reject requests that omit the `User-Agent` header or that use common defaults such as `'curl/7.79.1'`.
+
+In **undici**, the default user agent is `'undici'`. Since undici is integrated into Node.js core as the implementation of `fetch()`, requests made via `fetch()` use `'node'` as the default user agent.
+
+It is recommended to specify a **custom `User-Agent` header** when implementing crawlers. Providing a descriptive user agent allows servers to correctly identify the client and reduces the likelihood of requests being denied.
+
+A user agent string should include sufficient detail to identify the crawler and provide contact information. For example:
+
+```
+AcmeCo Crawler - acme.co - contact@acme.co
+```
+
+When adding contact details, avoid using personal identifiers such as your own name or a private email address—especially in a professional or employment context. Instead, use a role-based or organizational contact (e.g., crawler-team@company.com) to protect individual privacy while still enabling communication.
+
+If a crawler behaves unexpectedly—for example, due to misconfiguration or implementation errors—server administrators can use the information in the user agent to contact the operator and coordinate an appropriate resolution.
+
+The `User-Agent` header can be set on individual requests or applied globally by configuring a custom dispatcher.
+
+**Example: setting a `User-Agent` per request**
+
+```js
+import { fetch } from 'undici'
+
+const headers = {
+  'User-Agent': 'AcmeCo Crawler - acme.co - contact@acme.co'
+}
+
+const res = await fetch('https://example.com', { headers })
+```
+
+## Best Practices for Crawlers
+
+When developing a crawler, the following practices are recommended in addition to setting a descriptive `User-Agent` header:
+
+* **Respect `robots.txt`**
+  Follow the directives defined in the target site’s `robots.txt` file, including disallowed paths and optional crawl-delay settings (see [W3C guidelines](https://www.w3.org/wiki/Write_Web_Crawler)).
+
+* **Rate limiting**
+  Regulate request frequency to avoid imposing excessive load on servers. Introduce delays between requests or limit the number of concurrent requests. The W3C suggests at least one second between requests.
+
+* **Error handling**
+  Implement retry logic with exponential backoff for transient failures, and stop requests when persistent errors occur (e.g., HTTP 403 or 429).
+
+* **Monitoring and logging**
+  Track request volume, response codes, and error rates to detect misbehavior and address issues proactively.
+
+* **Contact information**
+  Always include valid and current contact details in the `User-Agent` string so that administrators can reach the crawler operator if necessary.
+
+## References and Further Reading
+
+* [RFC 9309: The Robots Exclusion Protocol](https://datatracker.ietf.org/doc/html/rfc9309)
+* [W3C Wiki: Write Web Crawler](https://www.w3.org/wiki/Write_Web_Crawler)
+* [Ethical Web Crawling (WWW 2010 Conference Paper)](https://archives.iw3c2.org/www2010/proceedings/www/p1101.pdf)