rock-req 5.0.3 → 5.1.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (3) hide show
  1. package/README.md +85 -48
  2. package/index.js +34 -24
  3. package/package.json +2 -2
package/README.md CHANGED
@@ -5,7 +5,8 @@
5
5
 
6
6
  <h1 align="center" style="border-bottom:none; font-size: 2.2em;">Rock-req.js</h1>
7
7
 
8
- <p align="center">Ensure your HTTP requests always reach their destination!</p>
8
+ <p align="center">⭐️⭐️ Ensure your HTTP requests always reach their destination as <b>efficiently</b> as possible! ⭐️⭐️</p>
9
+ <p align="center">Tested on Mac, Linux, Windows with NodeJS 16, 18, 19</p>
9
10
 
10
11
  [![npm][npm-image]][npm-url] [![ci][ci-image]][ci-url] [![javascript style guide][standard-image]][standard-url]
11
12
 
@@ -16,7 +17,7 @@
16
17
  [ci-url]: https://github.com/carboneio/rock-req/actions
17
18
  [ci-image]: https://img.shields.io/github/actions/workflow/status/carboneio/rock-req/ci.yml?branch=master
18
19
 
19
- ## 🔥 Why should you need this?
20
+ ## 🔥 Why?
20
21
 
21
22
  In most existing libraries (2023):
22
23
 
@@ -24,28 +25,57 @@ In most existing libraries (2023):
24
25
  - Using **multiple forward proxies** has several benefits like **higher availability** and **increased bandwidth** but
25
26
  **Intercepting retries** to use another Egress controller between two requests is not possible.
26
27
  - Many request libraries are heavy: node-fetch, superagent, needle, got, axios, request
27
- - Lightweight alternatives are not as light as they claim due to dependencies (simple-get, tiny-req, puny-req, ...)
28
+ - Lightweight alternatives are not as light as they claim due to dependencies (simple-get, tiny-req, puny-req, phin, ...)
28
29
 
29
- ⚡️ **Rock-req** solves these problems with only **160 lines of code** and **zero dependencies**
30
+ ⚡️ **Rock-req** solves these problems with only **150 lines of code** and **zero dependencies**
30
31
 
31
32
  It also supports many features:
32
33
 
33
34
  - Follows redirects
34
- - Handles gzip/deflate/brotli responses
35
+ - Handles **gzip/deflate/brotli** responses
35
36
  - Modify defaults
36
37
  - Extend and create new instances
37
- - Automatically destroy input/output stream on error (pipeline)
38
+ - Automatically destroy input/output **stream** on error and premature close event
39
+ - **Advanced retries**
40
+ - URL Rewrite
41
+ - **Ultra-fast (> 20k req/s)**
42
+ - Keep Alive by default (3000ms)
38
43
  - Composable
39
44
  - Timeouts
40
45
  - HTTPS / HTTP
41
46
  - Composes well with npm packages for features like cookies, proxies, form data, & OAuth
42
47
  - Keep 98% of the `simple-get` API (fork source)
43
48
 
44
- Like NodeJS pipeline, when the callback is called, the request is 100% finished, even with streams.
49
+ When the callback is called, the request is 100% finished, even with streams.
45
50
 
46
- ## Install
51
+
52
+ ## 🚀 Benchmark Rock-req vs got, axios, node-fetch, phin, simple-get, superagent, ...
53
+
54
+ Stop using "slow by-default" and "false-light" HTTP request libraries!
55
+
56
+
57
+ | Library | Speed | Size deps inc. |
58
+ | ------------ |-----------------:| --------------:|
59
+ | rock-req 🙋‍♂️ | 21797 req/s | 144 LOC |
60
+ | simple-get | 3260 req/s | 317 LOC |
61
+ | axios | 4910 req/s | 13983 LOC |
62
+ | got | 1762 req/s | 9227 LOC |
63
+ | fetch | 2102 req/s | 13334 LOC |
64
+ | request | 1869 req/s | 46572 LOC |
65
+ | superagent | 2100 req/s | 16109 LOC |
66
+ | phin | 1164 req/s | 331 LOC |
67
+ | undici* | 24378 req/s | 16225 LOC |
68
+
69
+
70
+ > `undici` is a low-level API, faster alternative to the native NodeJS http module. It is the glass ceiling limit for NodeJS.
71
+
72
+ > `rock-req` uses only the native NodeJS http module and provides many high-level features, a lot more than `phin` and `simple-get` with fewer lines of code.
73
+
74
+ > Tested with NodeJS 18.x LTS on Macbook Pro M1 Max
47
75
 
48
76
 
77
+ ## Install
78
+
49
79
  ```
50
80
  npm install rock-req
51
81
  ```
@@ -136,7 +166,7 @@ rock(opts, function (err, res, data) {} )
136
166
  **opts** can contain any value of NodeJS http.request with rock-req parameters. Here are the most used one:
137
167
 
138
168
  - `maxRedirects <number>`overwrite global maximum number of redirects. Defaults to 10
139
- - `maxRetry <number>` overwrite global maximum number of retries. Defaults to 0
169
+ - `maxRetry <number>` overwrite global maximum number of retries. Defaults to 1
140
170
  - `followRedirects <boolean>` do not follow redirects
141
171
  - `body <buffer> | <string> | <object> | <function>` body to post
142
172
  - `json <boolean>` automatically stringify/parse request/response Default : false
@@ -162,7 +192,8 @@ rock(opts, function (err, res, data) {} )
162
192
  Rock-req requires that input stream is initialized in a function.
163
193
 
164
194
  This function is invoked by rock-req for every request retry.
165
- If something goes wrong, the old stream is destroyed.
195
+
196
+ If something goes wrong, the Readable stream is destroyed automatically and the error can be captured with `'error'` event or `stream.finished` (optional).
166
197
 
167
198
  ```js
168
199
  const rock = require('rock-req')
@@ -191,33 +222,22 @@ Alternative syntax:
191
222
  ### Output Stream
192
223
 
193
224
  Rock-req requires that output stream is initialized in a function.
225
+
194
226
  This function is invoked by rock-req for every request retry.
195
227
 
228
+ If something goes wrong, the Writable stream is destroyed automatically and the error can be captured with `'error'` event or `stream.finished` (optional).
229
+
196
230
  ```js
197
231
  const rock = require('rock-req')
198
232
  const fs = require('fs')
199
- const { finished } = require('stream')
200
233
 
201
234
  // opts contains options passed in rock(opts). DO NOT MODIFY IT
202
235
  // res if the http response (res.statusCode, ...). DO NOT MODIFY IT and DO NOT CONSUME THE RES STREAM YOURSELF
203
236
  function createOutputStream(opts, res) {
204
237
 
205
238
  const writer = fs.createWriteStream('test_gfg.txt')
206
- // Internally, rock-req uses pipeline. If something goes wrong, the stream is destroyed automatically.
207
239
  // If you need to do some action (removing temporary files, ...), uses this native NodeJS method:
208
- const cleanup = finished(writer, (err) => {
209
- if (err) {
210
- // clean up things
211
- }
212
- // When using the finished() method in NodeJS, it's important to be aware that it can leave some event listeners
213
- // (specifically, the 'error', 'end', 'finish', and 'close' events) hanging around even after this callback function has been called.
214
- // This is intentional, as it helps prevent unexpected crashes if an error occurs due to incorrect stream implementations.
215
- // However, if you don't want these event listeners to stick around after the callback function has been called,
216
- // you can use the cleanup function that's returned by stream.finished() to remove them.
217
- // You'll need to explicitly call this cleanup function within your callback function to ensure that the event listeners get removed properly.
218
- cleanup();
219
- });
220
- // It must return a Writable stream. Otherwise, the request is cancel with an error
240
+ writer.on('error', (e) => { /* clean up your stuff */ })
221
241
  return writer
222
242
  }
223
243
 
@@ -230,7 +250,7 @@ rock(opts, function (err, res) {})
230
250
 
231
251
  ### Retry on failure
232
252
 
233
- By default, rock-req retries with the following errors if `maxRetry > 1`.
253
+ By default, rock-req retries with the following errors if `maxRetry > 0`.
234
254
 
235
255
  The callback is called when the request succeed or all retries are done
236
256
 
@@ -241,7 +261,6 @@ const rock = require('rock-req');
241
261
  rock.defaults.retryOnCode = [
242
262
  408, /* Request Timeout */
243
263
  429, /* Too Many Requests */
244
- 500, /* Internal Server Error */
245
264
  502, /* Bad Gateway */
246
265
  503, /* Service Unavailable */
247
266
  504, /* Gateway Timeout*/
@@ -263,24 +282,24 @@ rock.defaults.retryOnError = [
263
282
  const opts = {
264
283
  url : 'http://example.com',
265
284
  body : 'this is the POST body',
266
- maxRetry : 2 // 0 is the default value (= no retries)
285
+ maxRetry : 1
267
286
  }
268
287
  rock(opts, function (err, res, data) {} );
269
288
  ```
270
289
 
271
290
 
272
- ### Global options
291
+ ### Global options & Extend
273
292
 
274
293
  Change default parameters globally (not recommended), or create a new instance with specific paramaters (see below)
275
294
 
276
295
  ```js
277
296
  rock.defaults = {
278
- headers : { 'accept-encoding': 'gzip, deflate, br' },
279
- maxRedirects : 10,
280
- maxRetry : 0,
281
- retryDelay : 100, //ms
282
- retryOnCode : [408, 429, 500, 502, 503, 504, 521, 522, 524 ],
283
- retryOnError : ['ETIMEDOUT', 'ECONNRESET', 'EADDRINUSE', 'ECONNREFUSED','EPIPE', 'ENOTFOUND', 'ENETUNREACH', 'EAI_AGAIN' ],
297
+ headers : { 'accept-encoding': 'gzip, deflate, br' },
298
+ maxRedirects : 10,
299
+ maxRetry : 1,
300
+ retryDelay : 10, //ms
301
+ retryOnCode : [408, 429, 502, 503, 504, 521, 522, 524 ],
302
+ retryOnError : ['ETIMEDOUT', 'ECONNRESET', 'EADDRINUSE', 'ECONNREFUSED','EPIPE', 'ENOTFOUND', 'ENETUNREACH', 'EAI_AGAIN' ],
284
303
  // beforeRequest is called for each request, retry and redirect
285
304
  beforeRequest : (opts) => {
286
305
  // There options can be overwritted (= parsed opts.url)
@@ -294,6 +313,7 @@ rock.defaults = {
294
313
  opts.method = 'POST';
295
314
  opts.remainingRetry;
296
315
  opts.remainingRedirects;
316
+ opts.agent = otherHttpAgent;
297
317
 
298
318
  // READ-ONLY options (not exhaustive)
299
319
  opts.url; // DOT NOT OVERWRITE
@@ -308,20 +328,38 @@ rock.defaults = {
308
328
  }
309
329
  ```
310
330
 
311
- ### Extend and intercept retries
312
-
313
- Create a new instance with specific parameter instead of modifying `rock.defaults`
331
+ Create a new instance with specific parameters instead of modifying global `rock.defaults`.
314
332
 
315
333
  By default, this new instance inherits values of the instance source if options are not overwritten.
316
334
  Headers are merged. Then only the first level of the options object is merged (no deep travelling in sub-objects or arrays).
317
335
 
318
- Here is a basic example of `beforeRequest` interceptor to use [HAProxy as a forward proxy](https://www.haproxy.com/user-spotlight-series/haproxy-as-egress-controller/).
336
+ The `keepAliveDuration` can be changed only with `extend` method because `rock-req` creates new http Agent on new instances.
337
+
338
+ ```js
339
+ const myInstance = rock.extend({
340
+ keepAliveDuration : 0, // Change keep alive duration. Default to 3000ms. Set 0 to deactivate keep alive.
341
+ headers: {
342
+ 'Custom-header': 'x-for-proxy'
343
+ },
344
+ timeout : 1000
345
+ });
346
+
347
+ myInstance.get('http://example.com', function (err, res, data) {})
348
+ ```
349
+
350
+ ### Intercept retries for Higher Availability / Higher bandwidth
319
351
 
320
- `beforeRequest` is always called on each redirect/retry.
352
+ `beforeRequest` is always called on each request, each redirect and each retry.
321
353
  - on redirect, `opts.url` (and `hostname`, `port`, `protocol`, `path`) is updated to the new location. `opts.url` is null if it is a relative redirect.
322
354
  - on retry, `opts.url` (and `hostname`, `port`, `protocol`, `path`) have the same value as they did
323
355
  when the rock-req was initially called.
324
356
 
357
+ For example, you can dynamically change the http Agent to use a another proxy on each request.
358
+ Be careful, in this case, you must provide the right http/https Agent if there is a redirection from http to https.
359
+ Otherwise, rock-req automatically replaces your Agent with the correct one if the protocol changes after redirection.
360
+
361
+ Or, you can rewrite the URL if you want to use [HAProxy as a forward proxy](https://www.haproxy.com/user-spotlight-series/haproxy-as-egress-controller/).
362
+
325
363
 
326
364
  ```js
327
365
  const myInstance = rock.extend({
@@ -332,11 +370,7 @@ const myInstance = rock.extend({
332
370
  opts.port = 80;
333
371
  opts.path = `${hostname}/${port}${path}`;
334
372
  return opts;
335
- },
336
- headers: {
337
- 'Custom-header': 'x-for-proxy'
338
- },
339
- timeout : 1000
373
+ }
340
374
  });
341
375
 
342
376
  myInstance.get('http://example.com', function (err, res, data) {})
@@ -344,7 +378,6 @@ myInstance.get('http://example.com', function (err, res, data) {})
344
378
  ```
345
379
 
346
380
 
347
-
348
381
  ### Timeout
349
382
 
350
383
  You can set a timeout (in milliseconds) on the request with the `timeout` option.
@@ -557,12 +590,16 @@ Rock-req is a fork of [simple-get](https://github.com/feross/simple-get)
557
590
  - after `body = () => { const myStream = create(); return myStream; }`
558
591
 
559
592
 
560
- ## Notes:
593
+ ## TODO:
561
594
 
562
595
  - [ ] replace deprecated `url.parse` by `new URL` but new URL is slower than url.parse. Let's see if Node 20 LTS is faster
563
- - [ ] agent keep Alive
564
596
  - [ ] add advanced timeout (response timeout)
565
597
  - [ ] test prevError
598
+ - [ ] test HTTP abort signal option
599
+ - [ ] test input stream error with 502 error retry. Does stream.resume destroy all streams?
600
+ - [ ] promisify
601
+ - [ ] typescript type
602
+ - [ ] NodesJS 19 doesn't need agent.timeout to exit https://github.com/nodejs/node/issues/47228 https://github.com/nodejs/node/issues/2642
566
603
 
567
604
 
568
605
  # Supporters
package/index.js CHANGED
@@ -1,11 +1,9 @@
1
- module.exports = extend()
2
-
3
1
  const http = require('http')
4
2
  const https = require('https')
5
3
  const querystring = require('querystring')
6
4
  const url = require('url')
7
5
  const zlib = require('zlib')
8
- const { pipeline, Writable, Readable } = require('stream')
6
+ const { Writable } = require('stream')
9
7
 
10
8
  const isStream = o => o !== null && typeof o === 'object' && typeof o.pipe === 'function'
11
9
  const isFnStream = o => o instanceof Function
@@ -16,16 +14,17 @@ function extend (defaultOptions = {}) {
16
14
  let _default = {
17
15
  headers: { 'accept-encoding': 'gzip, deflate, br' },
18
16
  maxRedirects: 10,
19
- maxRetry: 0,
20
- retryDelay: 100, // ms
21
- retryOnCode: [408, 429, 500, 502, 503, 504, 521, 522, 524],
17
+ maxRetry: 1,
18
+ retryDelay: 10, // ms
19
+ keepAliveDuration: 3000, // ms
20
+ retryOnCode: [408, 429, 502, 503, 504, 521, 522, 524],
22
21
  retryOnError: ['ETIMEDOUT', 'ECONNRESET', 'EADDRINUSE', 'ECONNREFUSED', 'EPIPE', 'ENOTFOUND', 'ENETUNREACH', 'EAI_AGAIN'],
23
22
  beforeRequest: o => o
24
23
  }
25
24
  defaultOptions.headers = applyDefault(cloneLowerCase(defaultOptions.headers), _default.headers)
26
25
  _default = applyDefault(defaultOptions, _default) // inherits of parent options
26
+ const agents = [http, https].map(h => (_default.keepAliveDuration > 0) ? new h.Agent({ keepAlive: true, keepAliveMsecs: _default.keepAliveDuration, timeout: _default.keepAliveDuration /* remove free socket node < 19 */ }) : undefined)
27
27
 
28
- // all options https://nodejs.org/dist/latest-v18.x/docs/api/http.html#http_http_request_url_options_callback
29
28
  function rock (opts, directBody, cb) {
30
29
  if (typeof opts === 'string') opts = { url: opts }
31
30
  if (!cb) { cb = directBody } else { opts.body = directBody }
@@ -58,25 +57,39 @@ function extend (defaultOptions = {}) {
58
57
  if (opts.output && (isStream(opts.output) || !isFnStream(opts.output))) return cb(new Error('opts.output must be a function returning a Writable stream. RTFM'))
59
58
  if (opts.json) opts.headers.accept = 'application/json'
60
59
  if (opts.method) opts.method = opts.method.toUpperCase()
60
+ if (!opts.agent || opts.agent.protocol !== opts.protocol) opts.agent = (opts.protocol === 'https:' ? agents[1] : agents[0])
61
61
 
62
62
  const protocol = opts.protocol === 'https:' ? https : http // Support http/https urls
63
63
  const chunks = []
64
+ const streamToCleanOnError = []
64
65
  let requestAbortedOrEnded = false
65
66
  let response = null
66
67
  function onRequestEnd (err) {
67
68
  if (requestAbortedOrEnded === true) return
68
69
  requestAbortedOrEnded = true
69
- if (opts.retryOnError.indexOf(err?.code) !== -1 && --opts.remainingRetry > 0) {
70
- opts.prevError = err
71
- return setTimeout(rock, opts.retryDelay, opts, cb) // retry in 100ms
70
+ if (err) {
71
+ streamToCleanOnError.forEach(s => s.destroy(err))
72
+ if (opts.retryOnError.indexOf(err?.code) !== -1 && --opts.remainingRetry > 0) {
73
+ opts.prevError = err
74
+ return setTimeout(rock, opts.retryDelay, opts, cb) // retry in 100ms
75
+ }
76
+ return cb(err)
72
77
  }
73
- if (err) return cb(err)
74
78
  let data = Buffer.concat(chunks)
75
79
  if (opts.json) {
76
- try { data = JSON.parse(data.toString()) } catch (e) { return cb(e, response, data) }
80
+ try { data = data.length > 0 ? JSON.parse(data.toString()) : null } catch (e) { return cb(e, response, data) }
77
81
  }
78
82
  cb(null, response, data)
79
83
  }
84
+ function listen (stream, isLast = false) {
85
+ streamToCleanOnError.push(stream)
86
+ stream.once('error', onRequestEnd)
87
+ stream.once('close', () => {
88
+ if (stream.readableEnded === false || stream.writableEnded === false) return onRequestEnd(new Error('ERR_STREAM_PREMATURE_CLOSE'))
89
+ if (isLast === true) return onRequestEnd()
90
+ })
91
+ return stream
92
+ }
80
93
  const req = protocol.request(opts, res => {
81
94
  opts.prevStatusCode = res.statusCode
82
95
  // retry and leave
@@ -116,27 +129,22 @@ function extend (defaultOptions = {}) {
116
129
  const output = opts.output ? opts.output(opts, res) : new Writable({ write (chunk, enc, wcb) { chunks.push(chunk); wcb() } })
117
130
  switch (contentEncoding) {
118
131
  case 'br':
119
- pipeline(res, zlib.createBrotliDecompress(), output, onRequestEnd); break
132
+ listen(res).pipe(listen(zlib.createBrotliDecompress())).pipe(listen(output, true)); break
120
133
  case 'gzip':
121
134
  case 'deflate':
122
- pipeline(res, zlib.createUnzip(), output, onRequestEnd); break
135
+ listen(res).pipe(listen(zlib.createUnzip())).pipe(listen(output, true)); break
123
136
  default:
124
- pipeline(res, output, onRequestEnd); break
137
+ listen(res).pipe(listen(output, true)); break
125
138
  }
126
139
  })
127
140
  req.once('timeout', () => {
128
- // This timeout event can come after the input pipeline is finished (ex. timeout with no body)
129
141
  const _error = new Error('TimeoutError'); _error.code = 'ETIMEDOUT'
130
142
  req.destroy(_error) // we must destroy manually and send the error to the error listener to call onRequestEnd
131
143
  })
132
- req.once('error', (e) => {
133
- onRequestEnd(e) // error can happen before pipeline is executed when some interceptor are used such as nock
134
- req.destroy()
135
- })
136
- const _inputStream = isFnStream(body) ? body(opts) : Readable.from([body], { objectMode: false })
137
- pipeline(_inputStream, req, (e) => {
138
- if (e) onRequestEnd(e)
139
- })
144
+
145
+ if (isFnStream(body) === true) listen(body(opts)).pipe(listen(req))
146
+ else listen(req).end(body)
147
+
140
148
  return req
141
149
  }
142
150
 
@@ -155,3 +163,5 @@ function extend (defaultOptions = {}) {
155
163
  rock.extend = extend
156
164
  return rock
157
165
  }
166
+
167
+ module.exports = extend()
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "rock-req",
3
- "description": "Zero dependencies (160 LOC) & rock-solid request library: http/https, reliable retry on failure, redirects, gzip/deflate/brotli, extensible, proxy, streams, JSON mode, forms, timeout",
4
- "version": "5.0.3",
3
+ "description": "Ultra-light (150 LOC, No dependencies) & Ultra-fast request library with reliable retry on failure, http/https, redirects, gzip/deflate/brotli, extensible, proxy, streams, JSON mode, forms, timeout",
4
+ "version": "5.1.1",
5
5
  "author": {
6
6
  "name": "David Grelaud & Feross Aboukhadijeh"
7
7
  },