rock-req 5.0.3 → 5.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (3) hide show
  1. package/README.md +84 -48
  2. package/index.js +33 -23
  3. package/package.json +2 -2
package/README.md CHANGED
@@ -5,7 +5,7 @@
5
5
 
6
6
  <h1 align="center" style="border-bottom:none; font-size: 2.2em;">Rock-req.js</h1>
7
7
 
8
- <p align="center">Ensure your HTTP requests always reach their destination!</p>
8
+ <p align="center">⭐️⭐️ Ensure your HTTP requests always reach their destination as <b>efficiently</b> as possible! ⭐️⭐️</p>
9
9
 
10
10
  [![npm][npm-image]][npm-url] [![ci][ci-image]][ci-url] [![javascript style guide][standard-image]][standard-url]
11
11
 
@@ -16,7 +16,7 @@
16
16
  [ci-url]: https://github.com/carboneio/rock-req/actions
17
17
  [ci-image]: https://img.shields.io/github/actions/workflow/status/carboneio/rock-req/ci.yml?branch=master
18
18
 
19
- ## 🔥 Why should you need this?
19
+ ## 🔥 Why?
20
20
 
21
21
  In most existing libraries (2023):
22
22
 
@@ -24,28 +24,57 @@ In most existing libraries (2023):
24
24
  - Using **multiple forward proxies** has several benefits like **higher availability** and **increased bandwidth** but
25
25
  **Intercepting retries** to use another Egress controller between two requests is not possible.
26
26
  - Many request libraries are heavy: node-fetch, superagent, needle, got, axios, request
27
- - Lightweight alternatives are not as light as they claim due to dependencies (simple-get, tiny-req, puny-req, ...)
27
+ - Lightweight alternatives are not as light as they claim due to dependencies (simple-get, tiny-req, puny-req, phin, ...)
28
28
 
29
- ⚡️ **Rock-req** solves these problems with only **160 lines of code** and **zero dependencies**
29
+ ⚡️ **Rock-req** solves these problems with only **150 lines of code** and **zero dependencies**
30
30
 
31
31
  It also supports many features:
32
32
 
33
33
  - Follows redirects
34
- - Handles gzip/deflate/brotli responses
34
+ - Handles **gzip/deflate/brotli** responses
35
35
  - Modify defaults
36
36
  - Extend and create new instances
37
- - Automatically destroy input/output stream on error (pipeline)
37
+ - Automatically destroy input/output **stream** on error and premature close event
38
+ - **Advanced retries**
39
+ - URL Rewrite
40
+ - **Ultra-fast (> 20k req/s)**
41
+ - Keep Alive by default (3000ms)
38
42
  - Composable
39
43
  - Timeouts
40
44
  - HTTPS / HTTP
41
45
  - Composes well with npm packages for features like cookies, proxies, form data, & OAuth
42
46
  - Keep 98% of the `simple-get` API (fork source)
43
47
 
44
- Like NodeJS pipeline, when the callback is called, the request is 100% finished, even with streams.
48
+ When the callback is called, the request is 100% finished, even with streams.
45
49
 
46
- ## Install
50
+
51
+ ## 🚀 Benchmark Rock-req vs got, axios, node-fetch, phin, simple-get, superagent, ...
52
+
53
+ Stop using "slow by-default" and "false-light" HTTP request libraries!
54
+
55
+
56
+ | Library | Speed | Size deps inc. |
57
+ | ------------ |-----------------:| --------------:|
58
+ | rock-req 🙋‍♂️ | 21797 req/s | 144 LOC |
59
+ | simple-get | 3260 req/s | 317 LOC |
60
+ | axios | 4910 req/s | 13983 LOC |
61
+ | got | 1762 req/s | 9227 LOC |
62
+ | fetch | 2102 req/s | 13334 LOC |
63
+ | request | 1869 req/s | 46572 LOC |
64
+ | superagent | 2100 req/s | 16109 LOC |
65
+ | phin | 1164 req/s | 331 LOC |
66
+ | undici* | 24378 req/s | 16225 LOC |
67
+
68
+
69
+ > `undici` is a low-level API, faster alternative to the native NodeJS http module. It is the glass ceiling limit for NodeJS.
70
+
71
+ > `rock-req` uses only the native NodeJS http module and provides many high-level features, a lot more than `phin` and `simple-get` with fewer lines
72
+
73
+ > Tested with NodeJS 18.x LTS on Macbook Pro M1 Max
47
74
 
48
75
 
76
+ ## Install
77
+
49
78
  ```
50
79
  npm install rock-req
51
80
  ```
@@ -136,7 +165,7 @@ rock(opts, function (err, res, data) {} )
136
165
  **opts** can contain any value of NodeJS http.request with rock-req parameters. Here are the most used one:
137
166
 
138
167
  - `maxRedirects <number>`overwrite global maximum number of redirects. Defaults to 10
139
- - `maxRetry <number>` overwrite global maximum number of retries. Defaults to 0
168
+ - `maxRetry <number>` overwrite global maximum number of retries. Defaults to 1
140
169
  - `followRedirects <boolean>` do not follow redirects
141
170
  - `body <buffer> | <string> | <object> | <function>` body to post
142
171
  - `json <boolean>` automatically stringify/parse request/response Default : false
@@ -162,7 +191,8 @@ rock(opts, function (err, res, data) {} )
162
191
  Rock-req requires that input stream is initialized in a function.
163
192
 
164
193
  This function is invoked by rock-req for every request retry.
165
- If something goes wrong, the old stream is destroyed.
194
+
195
+ If something goes wrong, the Readable stream is destroyed automatically and the error can be captured with `'error'` event or `stream.finished` (optional).
166
196
 
167
197
  ```js
168
198
  const rock = require('rock-req')
@@ -191,33 +221,22 @@ Alternative syntax:
191
221
  ### Output Stream
192
222
 
193
223
  Rock-req requires that output stream is initialized in a function.
224
+
194
225
  This function is invoked by rock-req for every request retry.
195
226
 
227
+ If something goes wrong, the Writable stream is destroyed automatically and the error can be captured with `'error'` event or `stream.finished` (optional).
228
+
196
229
  ```js
197
230
  const rock = require('rock-req')
198
231
  const fs = require('fs')
199
- const { finished } = require('stream')
200
232
 
201
233
  // opts contains options passed in rock(opts). DO NOT MODIFY IT
202
234
  // res if the http response (res.statusCode, ...). DO NOT MODIFY IT and DO NOT CONSUME THE RES STREAM YOURSELF
203
235
  function createOutputStream(opts, res) {
204
236
 
205
237
  const writer = fs.createWriteStream('test_gfg.txt')
206
- // Internally, rock-req uses pipeline. If something goes wrong, the stream is destroyed automatically.
207
238
  // If you need to do some action (removing temporary files, ...), uses this native NodeJS method:
208
- const cleanup = finished(writer, (err) => {
209
- if (err) {
210
- // clean up things
211
- }
212
- // When using the finished() method in NodeJS, it's important to be aware that it can leave some event listeners
213
- // (specifically, the 'error', 'end', 'finish', and 'close' events) hanging around even after this callback function has been called.
214
- // This is intentional, as it helps prevent unexpected crashes if an error occurs due to incorrect stream implementations.
215
- // However, if you don't want these event listeners to stick around after the callback function has been called,
216
- // you can use the cleanup function that's returned by stream.finished() to remove them.
217
- // You'll need to explicitly call this cleanup function within your callback function to ensure that the event listeners get removed properly.
218
- cleanup();
219
- });
220
- // It must return a Writable stream. Otherwise, the request is cancel with an error
239
+ writer.on('error', (e) => { /* clean up your stuff */ })
221
240
  return writer
222
241
  }
223
242
 
@@ -230,7 +249,7 @@ rock(opts, function (err, res) {})
230
249
 
231
250
  ### Retry on failure
232
251
 
233
- By default, rock-req retries with the following errors if `maxRetry > 1`.
252
+ By default, rock-req retries with the following errors if `maxRetry > 0`.
234
253
 
235
254
  The callback is called when the request succeed or all retries are done
236
255
 
@@ -241,7 +260,6 @@ const rock = require('rock-req');
241
260
  rock.defaults.retryOnCode = [
242
261
  408, /* Request Timeout */
243
262
  429, /* Too Many Requests */
244
- 500, /* Internal Server Error */
245
263
  502, /* Bad Gateway */
246
264
  503, /* Service Unavailable */
247
265
  504, /* Gateway Timeout*/
@@ -263,24 +281,24 @@ rock.defaults.retryOnError = [
263
281
  const opts = {
264
282
  url : 'http://example.com',
265
283
  body : 'this is the POST body',
266
- maxRetry : 2 // 0 is the default value (= no retries)
284
+ maxRetry : 1
267
285
  }
268
286
  rock(opts, function (err, res, data) {} );
269
287
  ```
270
288
 
271
289
 
272
- ### Global options
290
+ ### Global options & Extend
273
291
 
274
292
  Change default parameters globally (not recommended), or create a new instance with specific paramaters (see below)
275
293
 
276
294
  ```js
277
295
  rock.defaults = {
278
- headers : { 'accept-encoding': 'gzip, deflate, br' },
279
- maxRedirects : 10,
280
- maxRetry : 0,
281
- retryDelay : 100, //ms
282
- retryOnCode : [408, 429, 500, 502, 503, 504, 521, 522, 524 ],
283
- retryOnError : ['ETIMEDOUT', 'ECONNRESET', 'EADDRINUSE', 'ECONNREFUSED','EPIPE', 'ENOTFOUND', 'ENETUNREACH', 'EAI_AGAIN' ],
296
+ headers : { 'accept-encoding': 'gzip, deflate, br' },
297
+ maxRedirects : 10,
298
+ maxRetry : 1,
299
+ retryDelay : 10, //ms
300
+ retryOnCode : [408, 429, 502, 503, 504, 521, 522, 524 ],
301
+ retryOnError : ['ETIMEDOUT', 'ECONNRESET', 'EADDRINUSE', 'ECONNREFUSED','EPIPE', 'ENOTFOUND', 'ENETUNREACH', 'EAI_AGAIN' ],
284
302
  // beforeRequest is called for each request, retry and redirect
285
303
  beforeRequest : (opts) => {
286
304
  // There options can be overwritted (= parsed opts.url)
@@ -294,6 +312,7 @@ rock.defaults = {
294
312
  opts.method = 'POST';
295
313
  opts.remainingRetry;
296
314
  opts.remainingRedirects;
315
+ opts.agent = otherHttpAgent;
297
316
 
298
317
  // READ-ONLY options (not exhaustive)
299
318
  opts.url; // DOT NOT OVERWRITE
@@ -308,20 +327,38 @@ rock.defaults = {
308
327
  }
309
328
  ```
310
329
 
311
- ### Extend and intercept retries
312
-
313
- Create a new instance with specific parameter instead of modifying `rock.defaults`
330
+ Create a new instance with specific parameters instead of modifying global `rock.defaults`.
314
331
 
315
332
  By default, this new instance inherits values of the instance source if options are not overwritten.
316
333
  Headers are merged. Then only the first level of the options object is merged (no deep travelling in sub-objects or arrays).
317
334
 
318
- Here is a basic example of `beforeRequest` interceptor to use [HAProxy as a forward proxy](https://www.haproxy.com/user-spotlight-series/haproxy-as-egress-controller/).
335
+ The `keepAliveDuration` can be changed only with `extend` method because `rock-req` creates new http Agent on new instances.
336
+
337
+ ```js
338
+ const myInstance = rock.extend({
339
+ keepAliveDuration : 0, // Change keep alive duration. Default to 3000ms. Set 0 to deactivate keep alive.
340
+ headers: {
341
+ 'Custom-header': 'x-for-proxy'
342
+ },
343
+ timeout : 1000
344
+ });
345
+
346
+ myInstance.get('http://example.com', function (err, res, data) {})
347
+ ```
348
+
349
+ ### Intercept retries for Higher Availability / Higher bandwidth
319
350
 
320
- `beforeRequest` is always called on each redirect/retry.
351
+ `beforeRequest` is always called on each request, each redirect and each retry.
321
352
  - on redirect, `opts.url` (and `hostname`, `port`, `protocol`, `path`) is updated to the new location. `opts.url` is null if it is a relative redirect.
322
353
  - on retry, `opts.url` (and `hostname`, `port`, `protocol`, `path`) have the same value as they did
323
354
  when the rock-req was initially called.
324
355
 
356
+ For example, you can dynamically change the http Agent to use a another proxy on each request.
357
+ Be careful, in this case, you must provide the right http/https Agent if there is a redirection from http to https.
358
+ Otherwise, rock-req automatically replaces your Agent with the correct one if the protocol changes after redirection.
359
+
360
+ Or, you can rewrite the URL if you want to use [HAProxy as a forward proxy](https://www.haproxy.com/user-spotlight-series/haproxy-as-egress-controller/).
361
+
325
362
 
326
363
  ```js
327
364
  const myInstance = rock.extend({
@@ -332,11 +369,7 @@ const myInstance = rock.extend({
332
369
  opts.port = 80;
333
370
  opts.path = `${hostname}/${port}${path}`;
334
371
  return opts;
335
- },
336
- headers: {
337
- 'Custom-header': 'x-for-proxy'
338
- },
339
- timeout : 1000
372
+ }
340
373
  });
341
374
 
342
375
  myInstance.get('http://example.com', function (err, res, data) {})
@@ -344,7 +377,6 @@ myInstance.get('http://example.com', function (err, res, data) {})
344
377
  ```
345
378
 
346
379
 
347
-
348
380
  ### Timeout
349
381
 
350
382
  You can set a timeout (in milliseconds) on the request with the `timeout` option.
@@ -557,12 +589,16 @@ Rock-req is a fork of [simple-get](https://github.com/feross/simple-get)
557
589
  - after `body = () => { const myStream = create(); return myStream; }`
558
590
 
559
591
 
560
- ## Notes:
592
+ ## TODO:
561
593
 
562
594
  - [ ] replace deprecated `url.parse` by `new URL` but new URL is slower than url.parse. Let's see if Node 20 LTS is faster
563
- - [ ] agent keep Alive
564
595
  - [ ] add advanced timeout (response timeout)
565
596
  - [ ] test prevError
597
+ - [ ] test HTTP abort signal option
598
+ - [ ] test input stream error with 502 error retry. Does stream.resume destroy all streams?
599
+ - [ ] promisify
600
+ - [ ] typescript type
601
+ - [ ] NodesJS 19 doesn't need agent.timeout to exit https://github.com/nodejs/node/issues/47228 https://github.com/nodejs/node/issues/2642
566
602
 
567
603
 
568
604
  # Supporters
package/index.js CHANGED
@@ -1,11 +1,9 @@
1
- module.exports = extend()
2
-
3
1
  const http = require('http')
4
2
  const https = require('https')
5
3
  const querystring = require('querystring')
6
4
  const url = require('url')
7
5
  const zlib = require('zlib')
8
- const { pipeline, Writable, Readable } = require('stream')
6
+ const { Writable } = require('stream')
9
7
 
10
8
  const isStream = o => o !== null && typeof o === 'object' && typeof o.pipe === 'function'
11
9
  const isFnStream = o => o instanceof Function
@@ -16,16 +14,17 @@ function extend (defaultOptions = {}) {
16
14
  let _default = {
17
15
  headers: { 'accept-encoding': 'gzip, deflate, br' },
18
16
  maxRedirects: 10,
19
- maxRetry: 0,
20
- retryDelay: 100, // ms
21
- retryOnCode: [408, 429, 500, 502, 503, 504, 521, 522, 524],
17
+ maxRetry: 1,
18
+ retryDelay: 10, // ms
19
+ keepAliveDuration: 3000, // ms
20
+ retryOnCode: [408, 429, 502, 503, 504, 521, 522, 524],
22
21
  retryOnError: ['ETIMEDOUT', 'ECONNRESET', 'EADDRINUSE', 'ECONNREFUSED', 'EPIPE', 'ENOTFOUND', 'ENETUNREACH', 'EAI_AGAIN'],
23
22
  beforeRequest: o => o
24
23
  }
25
24
  defaultOptions.headers = applyDefault(cloneLowerCase(defaultOptions.headers), _default.headers)
26
25
  _default = applyDefault(defaultOptions, _default) // inherits of parent options
26
+ const agents = [http, https].map(h => (_default.keepAliveDuration > 0) ? new h.Agent({ keepAlive: true, keepAliveMsecs: _default.keepAliveDuration, timeout: _default.keepAliveDuration /* remove free socket node < 19 */ }) : undefined)
27
27
 
28
- // all options https://nodejs.org/dist/latest-v18.x/docs/api/http.html#http_http_request_url_options_callback
29
28
  function rock (opts, directBody, cb) {
30
29
  if (typeof opts === 'string') opts = { url: opts }
31
30
  if (!cb) { cb = directBody } else { opts.body = directBody }
@@ -58,25 +57,39 @@ function extend (defaultOptions = {}) {
58
57
  if (opts.output && (isStream(opts.output) || !isFnStream(opts.output))) return cb(new Error('opts.output must be a function returning a Writable stream. RTFM'))
59
58
  if (opts.json) opts.headers.accept = 'application/json'
60
59
  if (opts.method) opts.method = opts.method.toUpperCase()
60
+ if (!opts.agent || opts.agent.protocol !== opts.protocol) opts.agent = (opts.protocol === 'https:' ? agents[1] : agents[0])
61
61
 
62
62
  const protocol = opts.protocol === 'https:' ? https : http // Support http/https urls
63
63
  const chunks = []
64
+ const streamToCleanOnError = []
64
65
  let requestAbortedOrEnded = false
65
66
  let response = null
66
67
  function onRequestEnd (err) {
67
68
  if (requestAbortedOrEnded === true) return
68
69
  requestAbortedOrEnded = true
69
- if (opts.retryOnError.indexOf(err?.code) !== -1 && --opts.remainingRetry > 0) {
70
- opts.prevError = err
71
- return setTimeout(rock, opts.retryDelay, opts, cb) // retry in 100ms
70
+ if (err) {
71
+ streamToCleanOnError.forEach(s => s.destroy(err))
72
+ if (opts.retryOnError.indexOf(err?.code) !== -1 && --opts.remainingRetry > 0) {
73
+ opts.prevError = err
74
+ return setTimeout(rock, opts.retryDelay, opts, cb) // retry in 100ms
75
+ }
76
+ return cb(err)
72
77
  }
73
- if (err) return cb(err)
74
78
  let data = Buffer.concat(chunks)
75
79
  if (opts.json) {
76
80
  try { data = JSON.parse(data.toString()) } catch (e) { return cb(e, response, data) }
77
81
  }
78
82
  cb(null, response, data)
79
83
  }
84
+ function listen (stream, isLast = false) {
85
+ streamToCleanOnError.push(stream)
86
+ stream.once('error', onRequestEnd)
87
+ stream.once('close', () => {
88
+ if (stream.readableEnded === false || stream.writableEnded === false) return onRequestEnd(new Error('ERR_STREAM_PREMATURE_CLOSE'))
89
+ if (isLast === true) return onRequestEnd()
90
+ })
91
+ return stream
92
+ }
80
93
  const req = protocol.request(opts, res => {
81
94
  opts.prevStatusCode = res.statusCode
82
95
  // retry and leave
@@ -116,27 +129,22 @@ function extend (defaultOptions = {}) {
116
129
  const output = opts.output ? opts.output(opts, res) : new Writable({ write (chunk, enc, wcb) { chunks.push(chunk); wcb() } })
117
130
  switch (contentEncoding) {
118
131
  case 'br':
119
- pipeline(res, zlib.createBrotliDecompress(), output, onRequestEnd); break
132
+ listen(res).pipe(listen(zlib.createBrotliDecompress())).pipe(listen(output, true)); break
120
133
  case 'gzip':
121
134
  case 'deflate':
122
- pipeline(res, zlib.createUnzip(), output, onRequestEnd); break
135
+ listen(res).pipe(listen(zlib.createUnzip())).pipe(listen(output, true)); break
123
136
  default:
124
- pipeline(res, output, onRequestEnd); break
137
+ listen(res).pipe(listen(output, true)); break
125
138
  }
126
139
  })
127
140
  req.once('timeout', () => {
128
- // This timeout event can come after the input pipeline is finished (ex. timeout with no body)
129
141
  const _error = new Error('TimeoutError'); _error.code = 'ETIMEDOUT'
130
142
  req.destroy(_error) // we must destroy manually and send the error to the error listener to call onRequestEnd
131
143
  })
132
- req.once('error', (e) => {
133
- onRequestEnd(e) // error can happen before pipeline is executed when some interceptor are used such as nock
134
- req.destroy()
135
- })
136
- const _inputStream = isFnStream(body) ? body(opts) : Readable.from([body], { objectMode: false })
137
- pipeline(_inputStream, req, (e) => {
138
- if (e) onRequestEnd(e)
139
- })
144
+
145
+ if (isFnStream(body) === true) listen(body(opts)).pipe(listen(req))
146
+ else listen(req).end(body)
147
+
140
148
  return req
141
149
  }
142
150
 
@@ -155,3 +163,5 @@ function extend (defaultOptions = {}) {
155
163
  rock.extend = extend
156
164
  return rock
157
165
  }
166
+
167
+ module.exports = extend()
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "rock-req",
3
- "description": "Zero dependencies (160 LOC) & rock-solid request library: http/https, reliable retry on failure, redirects, gzip/deflate/brotli, extensible, proxy, streams, JSON mode, forms, timeout",
4
- "version": "5.0.3",
3
+ "description": "Zero deps (160 LOC) & ultra-fast request library: http/https, reliable retry on failure, redirects, gzip/deflate/brotli, extensible, proxy, streams, JSON mode, forms, timeout",
4
+ "version": "5.1.0",
5
5
  "author": {
6
6
  "name": "David Grelaud & Feross Aboukhadijeh"
7
7
  },