rock-req 5.0.2 → 5.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (3) hide show
  1. package/README.md +84 -49
  2. package/index.js +33 -23
  3. package/package.json +2 -2
package/README.md CHANGED
@@ -5,7 +5,7 @@
5
5
 
6
6
  <h1 align="center" style="border-bottom:none; font-size: 2.2em;">Rock-req.js</h1>
7
7
 
8
- <p align="center">Ensure your HTTP requests always reach their destination!</p>
8
+ <p align="center">⭐️⭐️ Ensure your HTTP requests always reach their destination as <b>efficiently</b> as possible! ⭐️⭐️</p>
9
9
 
10
10
  [![npm][npm-image]][npm-url] [![ci][ci-image]][ci-url] [![javascript style guide][standard-image]][standard-url]
11
11
 
@@ -16,7 +16,7 @@
16
16
  [ci-url]: https://github.com/carboneio/rock-req/actions
17
17
  [ci-image]: https://img.shields.io/github/actions/workflow/status/carboneio/rock-req/ci.yml?branch=master
18
18
 
19
- ## 🔥 Why should you need this?
19
+ ## 🔥 Why?
20
20
 
21
21
  In most existing libraries (2023):
22
22
 
@@ -24,29 +24,57 @@ In most existing libraries (2023):
24
24
  - Using **multiple forward proxies** has several benefits like **higher availability** and **increased bandwidth** but
25
25
  **Intercepting retries** to use another Egress controller between two requests is not possible.
26
26
  - Many request libraries are heavy: node-fetch, superagent, needle, got, axios, request
27
- - Lightweight alternatives are not as light as they claim due to dependencies (simple-get, tiny-req, puny-req, ...)
27
+ - Lightweight alternatives are not as light as they claim due to dependencies (simple-get, tiny-req, puny-req, phin, ...)
28
28
 
29
- ⚡️ **Rock-req** solves these problems with only **160 lines of code** and **zero dependencies**
29
+ ⚡️ **Rock-req** solves these problems with only **150 lines of code** and **zero dependencies**
30
30
 
31
31
  It also supports many features:
32
32
 
33
33
  - Follows redirects
34
- - Handles gzip/deflate/brotli responses
34
+ - Handles **gzip/deflate/brotli** responses
35
35
  - Modify defaults
36
36
  - Extend and create new instances
37
- - Automatically destroy input/output stream on error (pipeline)
37
+ - Automatically destroy input/output **stream** on error and premature close event
38
+ - **Advanced retries**
39
+ - URL Rewrite
40
+ - **Ultra-fast (> 20k req/s)**
41
+ - Keep Alive by default (3000ms)
38
42
  - Composable
39
43
  - Timeouts
40
44
  - HTTPS / HTTP
41
45
  - Composes well with npm packages for features like cookies, proxies, form data, & OAuth
42
- - Typescript definition
43
46
  - Keep 98% of the `simple-get` API (fork source)
44
47
 
45
- Like NodeJS pipeline, when the callback is called, the request is 100% finished, even with streams.
48
+ When the callback is called, the request is 100% finished, even with streams.
46
49
 
47
- ## Install
50
+
51
+ ## 🚀 Benchmark Rock-req vs got, axios, node-fetch, phin, simple-get, superagent, ...
52
+
53
+ Stop using "slow by-default" and "false-light" HTTP request libraries!
54
+
55
+
56
+ | Library | Speed | Size deps inc. |
57
+ | ------------ |-----------------:| --------------:|
58
+ | rock-req 🙋‍♂️ | 21797 req/s | 144 LOC |
59
+ | simple-get | 3260 req/s | 317 LOC |
60
+ | axios | 4910 req/s | 13983 LOC |
61
+ | got | 1762 req/s | 9227 LOC |
62
+ | fetch | 2102 req/s | 13334 LOC |
63
+ | request | 1869 req/s | 46572 LOC |
64
+ | superagent | 2100 req/s | 16109 LOC |
65
+ | phin | 1164 req/s | 331 LOC |
66
+ | undici* | 24378 req/s | 16225 LOC |
67
+
68
+
69
+ > `undici` is a low-level API, faster alternative to the native NodeJS http module. It is the glass ceiling limit for NodeJS.
70
+
71
+ > `rock-req` uses only the native NodeJS http module and provides many high-level features, a lot more than `phin` and `simple-get` with fewer lines
72
+
73
+ > Tested with NodeJS 18.x LTS on Macbook Pro M1 Max
48
74
 
49
75
 
76
+ ## Install
77
+
50
78
  ```
51
79
  npm install rock-req
52
80
  ```
@@ -137,7 +165,7 @@ rock(opts, function (err, res, data) {} )
137
165
  **opts** can contain any value of NodeJS http.request with rock-req parameters. Here are the most used one:
138
166
 
139
167
  - `maxRedirects <number>`overwrite global maximum number of redirects. Defaults to 10
140
- - `maxRetry <number>` overwrite global maximum number of retries. Defaults to 0
168
+ - `maxRetry <number>` overwrite global maximum number of retries. Defaults to 1
141
169
  - `followRedirects <boolean>` do not follow redirects
142
170
  - `body <buffer> | <string> | <object> | <function>` body to post
143
171
  - `json <boolean>` automatically stringify/parse request/response Default : false
@@ -163,7 +191,8 @@ rock(opts, function (err, res, data) {} )
163
191
  Rock-req requires that input stream is initialized in a function.
164
192
 
165
193
  This function is invoked by rock-req for every request retry.
166
- If something goes wrong, the old stream is destroyed.
194
+
195
+ If something goes wrong, the Readable stream is destroyed automatically and the error can be captured with `'error'` event or `stream.finished` (optional).
167
196
 
168
197
  ```js
169
198
  const rock = require('rock-req')
@@ -192,33 +221,22 @@ Alternative syntax:
192
221
  ### Output Stream
193
222
 
194
223
  Rock-req requires that output stream is initialized in a function.
224
+
195
225
  This function is invoked by rock-req for every request retry.
196
226
 
227
+ If something goes wrong, the Writable stream is destroyed automatically and the error can be captured with `'error'` event or `stream.finished` (optional).
228
+
197
229
  ```js
198
230
  const rock = require('rock-req')
199
231
  const fs = require('fs')
200
- const { finished } = require('stream')
201
232
 
202
233
  // opts contains options passed in rock(opts). DO NOT MODIFY IT
203
234
  // res if the http response (res.statusCode, ...). DO NOT MODIFY IT and DO NOT CONSUME THE RES STREAM YOURSELF
204
235
  function createOutputStream(opts, res) {
205
236
 
206
237
  const writer = fs.createWriteStream('test_gfg.txt')
207
- // Internally, rock-req uses pipeline. If something goes wrong, the stream is destroyed automatically.
208
238
  // If you need to do some action (removing temporary files, ...), uses this native NodeJS method:
209
- const cleanup = finished(writer, (err) => {
210
- if (err) {
211
- // clean up things
212
- }
213
- // When using the finished() method in NodeJS, it's important to be aware that it can leave some event listeners
214
- // (specifically, the 'error', 'end', 'finish', and 'close' events) hanging around even after this callback function has been called.
215
- // This is intentional, as it helps prevent unexpected crashes if an error occurs due to incorrect stream implementations.
216
- // However, if you don't want these event listeners to stick around after the callback function has been called,
217
- // you can use the cleanup function that's returned by stream.finished() to remove them.
218
- // You'll need to explicitly call this cleanup function within your callback function to ensure that the event listeners get removed properly.
219
- cleanup();
220
- });
221
- // It must return a Writable stream. Otherwise, the request is cancel with an error
239
+ writer.on('error', (e) => { /* clean up your stuff */ })
222
240
  return writer
223
241
  }
224
242
 
@@ -231,7 +249,7 @@ rock(opts, function (err, res) {})
231
249
 
232
250
  ### Retry on failure
233
251
 
234
- By default, rock-req retries with the following errors if `maxRetry > 1`.
252
+ By default, rock-req retries with the following errors if `maxRetry > 0`.
235
253
 
236
254
  The callback is called when the request succeed or all retries are done
237
255
 
@@ -242,7 +260,6 @@ const rock = require('rock-req');
242
260
  rock.defaults.retryOnCode = [
243
261
  408, /* Request Timeout */
244
262
  429, /* Too Many Requests */
245
- 500, /* Internal Server Error */
246
263
  502, /* Bad Gateway */
247
264
  503, /* Service Unavailable */
248
265
  504, /* Gateway Timeout*/
@@ -264,24 +281,24 @@ rock.defaults.retryOnError = [
264
281
  const opts = {
265
282
  url : 'http://example.com',
266
283
  body : 'this is the POST body',
267
- maxRetry : 2 // 0 is the default value (= no retries)
284
+ maxRetry : 1
268
285
  }
269
286
  rock(opts, function (err, res, data) {} );
270
287
  ```
271
288
 
272
289
 
273
- ### Global options
290
+ ### Global options & Extend
274
291
 
275
292
  Change default parameters globally (not recommended), or create a new instance with specific paramaters (see below)
276
293
 
277
294
  ```js
278
295
  rock.defaults = {
279
- headers : { 'accept-encoding': 'gzip, deflate, br' },
280
- maxRedirects : 10,
281
- maxRetry : 0,
282
- retryDelay : 100, //ms
283
- retryOnCode : [408, 429, 500, 502, 503, 504, 521, 522, 524 ],
284
- retryOnError : ['ETIMEDOUT', 'ECONNRESET', 'EADDRINUSE', 'ECONNREFUSED','EPIPE', 'ENOTFOUND', 'ENETUNREACH', 'EAI_AGAIN' ],
296
+ headers : { 'accept-encoding': 'gzip, deflate, br' },
297
+ maxRedirects : 10,
298
+ maxRetry : 1,
299
+ retryDelay : 10, //ms
300
+ retryOnCode : [408, 429, 502, 503, 504, 521, 522, 524 ],
301
+ retryOnError : ['ETIMEDOUT', 'ECONNRESET', 'EADDRINUSE', 'ECONNREFUSED','EPIPE', 'ENOTFOUND', 'ENETUNREACH', 'EAI_AGAIN' ],
285
302
  // beforeRequest is called for each request, retry and redirect
286
303
  beforeRequest : (opts) => {
287
304
  // There options can be overwritted (= parsed opts.url)
@@ -295,6 +312,7 @@ rock.defaults = {
295
312
  opts.method = 'POST';
296
313
  opts.remainingRetry;
297
314
  opts.remainingRedirects;
315
+ opts.agent = otherHttpAgent;
298
316
 
299
317
  // READ-ONLY options (not exhaustive)
300
318
  opts.url; // DOT NOT OVERWRITE
@@ -309,20 +327,38 @@ rock.defaults = {
309
327
  }
310
328
  ```
311
329
 
312
- ### Extend and intercept retries
313
-
314
- Create a new instance with specific parameter instead of modifying `rock.defaults`
330
+ Create a new instance with specific parameters instead of modifying global `rock.defaults`.
315
331
 
316
332
  By default, this new instance inherits values of the instance source if options are not overwritten.
317
333
  Headers are merged. Then only the first level of the options object is merged (no deep travelling in sub-objects or arrays).
318
334
 
319
- Here is a basic example of `beforeRequest` interceptor to use [HAProxy as a forward proxy](https://www.haproxy.com/user-spotlight-series/haproxy-as-egress-controller/).
335
+ The `keepAliveDuration` can be changed only with `extend` method because `rock-req` creates new http Agent on new instances.
336
+
337
+ ```js
338
+ const myInstance = rock.extend({
339
+ keepAliveDuration : 0, // Change keep alive duration. Default to 3000ms. Set 0 to deactivate keep alive.
340
+ headers: {
341
+ 'Custom-header': 'x-for-proxy'
342
+ },
343
+ timeout : 1000
344
+ });
345
+
346
+ myInstance.get('http://example.com', function (err, res, data) {})
347
+ ```
348
+
349
+ ### Intercept retries for Higher Availability / Higher bandwidth
320
350
 
321
- `beforeRequest` is always called on each redirect/retry.
351
+ `beforeRequest` is always called on each request, each redirect and each retry.
322
352
  - on redirect, `opts.url` (and `hostname`, `port`, `protocol`, `path`) is updated to the new location. `opts.url` is null if it is a relative redirect.
323
353
  - on retry, `opts.url` (and `hostname`, `port`, `protocol`, `path`) have the same value as they did
324
354
  when the rock-req was initially called.
325
355
 
356
+ For example, you can dynamically change the http Agent to use a another proxy on each request.
357
+ Be careful, in this case, you must provide the right http/https Agent if there is a redirection from http to https.
358
+ Otherwise, rock-req automatically replaces your Agent with the correct one if the protocol changes after redirection.
359
+
360
+ Or, you can rewrite the URL if you want to use [HAProxy as a forward proxy](https://www.haproxy.com/user-spotlight-series/haproxy-as-egress-controller/).
361
+
326
362
 
327
363
  ```js
328
364
  const myInstance = rock.extend({
@@ -333,11 +369,7 @@ const myInstance = rock.extend({
333
369
  opts.port = 80;
334
370
  opts.path = `${hostname}/${port}${path}`;
335
371
  return opts;
336
- },
337
- headers: {
338
- 'Custom-header': 'x-for-proxy'
339
- },
340
- timeout : 1000
372
+ }
341
373
  });
342
374
 
343
375
  myInstance.get('http://example.com', function (err, res, data) {})
@@ -345,7 +377,6 @@ myInstance.get('http://example.com', function (err, res, data) {})
345
377
  ```
346
378
 
347
379
 
348
-
349
380
  ### Timeout
350
381
 
351
382
  You can set a timeout (in milliseconds) on the request with the `timeout` option.
@@ -558,12 +589,16 @@ Rock-req is a fork of [simple-get](https://github.com/feross/simple-get)
558
589
  - after `body = () => { const myStream = create(); return myStream; }`
559
590
 
560
591
 
561
- ## Notes:
592
+ ## TODO:
562
593
 
563
594
  - [ ] replace deprecated `url.parse` by `new URL` but new URL is slower than url.parse. Let's see if Node 20 LTS is faster
564
- - [ ] agent keep Alive
565
595
  - [ ] add advanced timeout (response timeout)
566
596
  - [ ] test prevError
597
+ - [ ] test HTTP abort signal option
598
+ - [ ] test input stream error with 502 error retry. Does stream.resume destroy all streams?
599
+ - [ ] promisify
600
+ - [ ] typescript type
601
+ - [ ] NodesJS 19 doesn't need agent.timeout to exit https://github.com/nodejs/node/issues/47228 https://github.com/nodejs/node/issues/2642
567
602
 
568
603
 
569
604
  # Supporters
package/index.js CHANGED
@@ -1,11 +1,9 @@
1
- module.exports = extend()
2
-
3
1
  const http = require('http')
4
2
  const https = require('https')
5
3
  const querystring = require('querystring')
6
4
  const url = require('url')
7
5
  const zlib = require('zlib')
8
- const { pipeline, Writable, Readable } = require('stream')
6
+ const { Writable } = require('stream')
9
7
 
10
8
  const isStream = o => o !== null && typeof o === 'object' && typeof o.pipe === 'function'
11
9
  const isFnStream = o => o instanceof Function
@@ -16,16 +14,17 @@ function extend (defaultOptions = {}) {
16
14
  let _default = {
17
15
  headers: { 'accept-encoding': 'gzip, deflate, br' },
18
16
  maxRedirects: 10,
19
- maxRetry: 0,
20
- retryDelay: 100, // ms
21
- retryOnCode: [408, 429, 500, 502, 503, 504, 521, 522, 524],
17
+ maxRetry: 1,
18
+ retryDelay: 10, // ms
19
+ keepAliveDuration: 3000, // ms
20
+ retryOnCode: [408, 429, 502, 503, 504, 521, 522, 524],
22
21
  retryOnError: ['ETIMEDOUT', 'ECONNRESET', 'EADDRINUSE', 'ECONNREFUSED', 'EPIPE', 'ENOTFOUND', 'ENETUNREACH', 'EAI_AGAIN'],
23
22
  beforeRequest: o => o
24
23
  }
25
24
  defaultOptions.headers = applyDefault(cloneLowerCase(defaultOptions.headers), _default.headers)
26
25
  _default = applyDefault(defaultOptions, _default) // inherits of parent options
26
+ const agents = [http, https].map(h => (_default.keepAliveDuration > 0) ? new h.Agent({ keepAlive: true, keepAliveMsecs: _default.keepAliveDuration, timeout: _default.keepAliveDuration /* remove free socket node < 19 */ }) : undefined)
27
27
 
28
- // all options https://nodejs.org/dist/latest-v18.x/docs/api/http.html#http_http_request_url_options_callback
29
28
  function rock (opts, directBody, cb) {
30
29
  if (typeof opts === 'string') opts = { url: opts }
31
30
  if (!cb) { cb = directBody } else { opts.body = directBody }
@@ -58,25 +57,39 @@ function extend (defaultOptions = {}) {
58
57
  if (opts.output && (isStream(opts.output) || !isFnStream(opts.output))) return cb(new Error('opts.output must be a function returning a Writable stream. RTFM'))
59
58
  if (opts.json) opts.headers.accept = 'application/json'
60
59
  if (opts.method) opts.method = opts.method.toUpperCase()
60
+ if (!opts.agent || opts.agent.protocol !== opts.protocol) opts.agent = (opts.protocol === 'https:' ? agents[1] : agents[0])
61
61
 
62
62
  const protocol = opts.protocol === 'https:' ? https : http // Support http/https urls
63
63
  const chunks = []
64
+ const streamToCleanOnError = []
64
65
  let requestAbortedOrEnded = false
65
66
  let response = null
66
67
  function onRequestEnd (err) {
67
68
  if (requestAbortedOrEnded === true) return
68
69
  requestAbortedOrEnded = true
69
- if (opts.retryOnError.indexOf(err?.code) !== -1 && --opts.remainingRetry > 0) {
70
- opts.prevError = err
71
- return setTimeout(rock, opts.retryDelay, opts, cb) // retry in 100ms
70
+ if (err) {
71
+ streamToCleanOnError.forEach(s => s.destroy(err))
72
+ if (opts.retryOnError.indexOf(err?.code) !== -1 && --opts.remainingRetry > 0) {
73
+ opts.prevError = err
74
+ return setTimeout(rock, opts.retryDelay, opts, cb) // retry in 100ms
75
+ }
76
+ return cb(err)
72
77
  }
73
- if (err) return cb(err)
74
78
  let data = Buffer.concat(chunks)
75
79
  if (opts.json) {
76
80
  try { data = JSON.parse(data.toString()) } catch (e) { return cb(e, response, data) }
77
81
  }
78
82
  cb(null, response, data)
79
83
  }
84
+ function listen (stream, isLast = false) {
85
+ streamToCleanOnError.push(stream)
86
+ stream.once('error', onRequestEnd)
87
+ stream.once('close', () => {
88
+ if (stream.readableEnded === false || stream.writableEnded === false) return onRequestEnd(new Error('ERR_STREAM_PREMATURE_CLOSE'))
89
+ if (isLast === true) return onRequestEnd()
90
+ })
91
+ return stream
92
+ }
80
93
  const req = protocol.request(opts, res => {
81
94
  opts.prevStatusCode = res.statusCode
82
95
  // retry and leave
@@ -116,27 +129,22 @@ function extend (defaultOptions = {}) {
116
129
  const output = opts.output ? opts.output(opts, res) : new Writable({ write (chunk, enc, wcb) { chunks.push(chunk); wcb() } })
117
130
  switch (contentEncoding) {
118
131
  case 'br':
119
- pipeline(res, zlib.createBrotliDecompress(), output, onRequestEnd); break
132
+ listen(res).pipe(listen(zlib.createBrotliDecompress())).pipe(listen(output, true)); break
120
133
  case 'gzip':
121
134
  case 'deflate':
122
- pipeline(res, zlib.createUnzip(), output, onRequestEnd); break
135
+ listen(res).pipe(listen(zlib.createUnzip())).pipe(listen(output, true)); break
123
136
  default:
124
- pipeline(res, output, onRequestEnd); break
137
+ listen(res).pipe(listen(output, true)); break
125
138
  }
126
139
  })
127
140
  req.once('timeout', () => {
128
- // This timeout event can come after the input pipeline is finished (ex. timeout with no body)
129
141
  const _error = new Error('TimeoutError'); _error.code = 'ETIMEDOUT'
130
142
  req.destroy(_error) // we must destroy manually and send the error to the error listener to call onRequestEnd
131
143
  })
132
- req.once('error', (e) => {
133
- onRequestEnd(e) // error can happen before pipeline is executed when some interceptor are used such as nock
134
- req.destroy()
135
- })
136
- const _inputStream = isFnStream(body) ? body(opts) : Readable.from([body], { objectMode: false })
137
- pipeline(_inputStream, req, (e) => {
138
- if (e) onRequestEnd(e)
139
- })
144
+
145
+ if (isFnStream(body) === true) listen(body(opts)).pipe(listen(req))
146
+ else listen(req).end(body)
147
+
140
148
  return req
141
149
  }
142
150
 
@@ -155,3 +163,5 @@ function extend (defaultOptions = {}) {
155
163
  rock.extend = extend
156
164
  return rock
157
165
  }
166
+
167
+ module.exports = extend()
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "rock-req",
3
- "description": "Zero dependencies (160 LOC) & rock-solid request library: http/https, reliable retry on failure, redirects, gzip/deflate/brotli, extensible, proxy, streams, JSON mode, forms, timeout",
4
- "version": "5.0.2",
3
+ "description": "Zero deps (160 LOC) & ultra-fast request library: http/https, reliable retry on failure, redirects, gzip/deflate/brotli, extensible, proxy, streams, JSON mode, forms, timeout",
4
+ "version": "5.1.0",
5
5
  "author": {
6
6
  "name": "David Grelaud & Feross Aboukhadijeh"
7
7
  },