rock-req 5.1.2 → 5.1.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (3) hide show
  1. package/README.md +4 -519
  2. package/index.js +1 -1
  3. package/package.json +1 -1
package/README.md CHANGED
@@ -78,527 +78,15 @@ Stop using "slow by-default" and "false-light" HTTP request libraries!
78
78
 
79
79
  ## Install
80
80
 
81
- ```
81
+ ```bash
82
82
  npm install rock-req
83
83
  ```
84
84
 
85
- ## Usage
86
-
87
- All functions accept two or three parameters:
88
-
89
- ```js
90
- const rock = require('rock-req')
91
- rock(optsOrURL [, bodyOrStreamFn], callback)
92
- ````
93
-
94
- - `optsOrURL` can be an object or a the URL
95
- - `bodyOrStreamFn` can be a buffer/string/object or a function returning an input stream for sending the body of POST/PUT/PATCH/DELETE requests
96
- - `callback(err, res, data)` called only when everything is finished (even with streams).
97
-
98
-
99
- ### GET, HEAD requests
100
-
101
- ```js
102
- // res is the server response, already consumed by rock-req. The result is in data
103
- rock.get('http://ex.com', (err, res, data) => {
104
- console.log(res.statusCode) // 200
105
- console.log(data) // Buffer('server response')
106
- })
107
- ```
108
-
109
- Alternative syntax:
110
-
111
- ```js
112
- rock({ method: 'GET', url: 'http://ex.com' }, function (err, res, data) {} )
113
- // OR
114
- rock.concat({ method: 'GET', url: 'http://ex.com' }, function (err, res, data) {} )
115
- ```
116
-
117
- Head requests:
118
-
119
- ```js
120
- rock.head('http://example.com', (err, res, data) => {})
121
- ```
122
-
123
- ### POST, PUT, PATCH, DELETE requests
124
-
125
- Use the second paramater to pass the body:
126
-
127
- ```js
128
- rock.post('http://ex.com', 'POST body', (err, res, data) => {})
129
- ```
130
-
131
- Alternative syntax:
132
-
133
- ```js
134
- rock({ method: 'POST', url : 'http://ex.com', body : 'POST body' }, function (err, res, data) {} )
135
- ```
136
-
137
- ### JSON request shortcuts
138
-
139
- Automatically serialize/deserialize request and response with JSON with `getJSON`, `putJSON`, `postJSON`, `deleteJSON`, ...
140
-
141
- ```js
142
- rock.putJSON('http://ex.com', { id : 123 }, (err, res, data) => {
143
- console.log(data) // already JSON.parsed
144
- })
145
- ```
146
-
147
- Alternative syntax:
148
-
149
- ```js
150
- rock({ method: 'PUT', url: 'http://ex.com', body: { id : 123 }, json: true }, function (err, res, data) {} )
151
- ```
152
-
153
- ### All options:
154
-
155
-
156
- ```js
157
- const opts = {
158
- url : 'http://example.com',
159
- method : 'POST',
160
- body : 'this is the POST body',
161
- headers: {
162
- 'user-agent': 'my cool app'
163
- }
164
- }
165
- rock(opts, function (err, res, data) {} )
166
- ```
167
-
168
- **opts** can contain any value of NodeJS http.request with rock-req parameters. Here are the most used one:
169
-
170
- - `maxRedirects <number>`overwrite global maximum number of redirects. Defaults to 10
171
- - `maxRetry <number>` overwrite global maximum number of retries. Defaults to 1
172
- - `followRedirects <boolean>` do not follow redirects
173
- - `body <buffer> | <string> | <object> | <function>` body to post
174
- - `json <boolean>` automatically stringify/parse request/response Default : false
175
- - `url <string>` the destination URL
176
- - `method <string>` A string specifying the HTTP request method. Default: 'GET'.
177
- - `headers <object>` An object containing request headers. Default: 'accept-encoding': 'gzip, deflate, br'
178
- - `timeout <number>`: A number specifying the socket timeout in milliseconds.
179
- - `auth <string>` Basic authentication ('user:password') to compute an Authorization header.
180
- - `port <number>` Port of remote server. Default: defaultPort if set, else 80.
181
- - `host <string>` A domain name or IP address of the server to issue the request to. Default: 'localhost'.
182
- - `hostname <string>` Alias for host.
183
- - `path <string>` Request path. Should include query string if any. E.G. '/index.html?page=12'.
184
- - `protocol <string>` Protocol to use. Default: 'http:'.
185
- - `setHost <boolean>`: Specifies whether or not to automatically add the Host header. Defaults to true.
186
- - `agent <http.Agent> | <boolean>` Controls Agent behavior. Possible values:
187
- - `undefined` (default): use http.globalAgent for this host and port.
188
- - `Agent object`: explicitly use the passed in Agent.
189
- - `false` causes a new Agent with default values to be used.
190
-
191
-
192
- ### Input Stream
193
-
194
- Rock-req requires that input stream is initialized in a function.
195
-
196
- This function is invoked by rock-req for every request retry.
197
-
198
- If something goes wrong, the Readable stream is destroyed automatically and the error can be captured with `'error'` event or `stream.finished` (optional).
199
-
200
- ```js
201
- const rock = require('rock-req')
202
- const fs = require('fs')
203
-
204
- // opts contains options passed in rock(opts). DO NOT MODIFY IT
205
- function createInputStream(opts) {
206
- return fs.createReadStream('input.txt');
207
- }
208
-
209
- const opts = {
210
- url : 'http://example.com',
211
- body: createInputStream
212
- }
213
- rock(opts, function (err, res, data) {})
214
- ```
215
-
216
- Alternative syntax:
217
-
218
- ```js
219
- rock.post('http://example.com', createInputStream, function (err, res, data) {})
220
- // or with more object options:
221
- rock.post(opts, createInputStream, function (err, res, data) {})
222
- ```
223
-
224
- ### Output Stream
225
-
226
- Rock-req requires that output stream is initialized in a function.
227
-
228
- This function is invoked by rock-req for every request retry.
229
-
230
- If something goes wrong, the Writable stream is destroyed automatically and the error can be captured with `'error'` event or `stream.finished` (optional).
231
-
232
- ```js
233
- const rock = require('rock-req')
234
- const fs = require('fs')
235
-
236
- // opts contains options passed in rock(opts). DO NOT MODIFY IT
237
- // res if the http response (res.statusCode, ...). DO NOT MODIFY IT and DO NOT CONSUME THE RES STREAM YOURSELF
238
- function createOutputStream(opts, res) {
239
-
240
- const writer = fs.createWriteStream('test_gfg.txt')
241
- // If you need to do some action (removing temporary files, ...), uses this native NodeJS method:
242
- writer.on('error', (e) => { /* clean up your stuff */ })
243
- return writer
244
- }
245
-
246
- const opts = {
247
- url : 'http://example.com',
248
- output : createOutputStream
249
- }
250
- rock(opts, function (err, res) {})
251
- ```
252
-
253
- ### Retry on failure
254
-
255
- By default, rock-req retries with the following errors if `maxRetry > 0`.
256
-
257
- The callback is called when the request succeed or all retries are done
258
-
259
- ```js
260
- const rock = require('rock-req');
261
-
262
- // default values can be overwritten like this:
263
- rock.defaults.retryOnCode = [
264
- 408, /* Request Timeout */
265
- 429, /* Too Many Requests */
266
- 502, /* Bad Gateway */
267
- 503, /* Service Unavailable */
268
- 504, /* Gateway Timeout*/
269
- 521, /* Web Server Is Down*/
270
- 522, /* Cloudflare Connection Timed Out */
271
- 524 /* Cloudflare A Timeout Occurred */
272
- ];
273
- rock.defaults.retryOnError = [
274
- 'ETIMEDOUT', /* One of the timeout limits was reached. */
275
- 'ECONNRESET', /* The connection was forcibly closed. */
276
- 'EADDRINUSE', /* Could not bind to any free port */
277
- 'ECONNREFUSED', /* The connection was refused by the server. */
278
- 'EPIPE', /* The remote side of the stream being written has been closed. */
279
- 'ENOTFOUND', /* Could not resolve the hostname to an IP address. */
280
- 'ENETUNREACH', /* No internet connection. */
281
- 'EAI_AGAIN' /* DNS lookup timed out. */
282
- ];
283
-
284
- const opts = {
285
- url : 'http://example.com',
286
- body : 'this is the POST body',
287
- maxRetry : 1
288
- }
289
- rock(opts, function (err, res, data) {} );
290
- ```
291
-
292
-
293
- ### Global options & Extend
294
-
295
- Change default parameters globally (not recommended), or create a new instance with specific paramaters (see below)
296
-
297
- ```js
298
- rock.defaults = {
299
- headers : { 'accept-encoding': 'gzip, deflate, br' },
300
- maxRedirects : 10,
301
- maxRetry : 1,
302
- retryDelay : 10, //ms
303
- retryOnCode : [408, 429, 502, 503, 504, 521, 522, 524 ],
304
- retryOnError : ['ETIMEDOUT', 'ECONNRESET', 'EADDRINUSE', 'ECONNREFUSED','EPIPE', 'ENOTFOUND', 'ENETUNREACH', 'EAI_AGAIN' ],
305
- // beforeRequest is called for each request, retry and redirect
306
- beforeRequest : (opts) => {
307
- // There options can be overwritted (= parsed opts.url)
308
- opts.protocol = 'https:' // or 'http:'
309
- opts.hostname = 'google.com';
310
- opts.port = 443;
311
- opts.path = '/mypage.html?bla=1#hash';
312
- opts.auth = '';
313
- opts.headers = {};
314
- opts.body = {};
315
- opts.method = 'POST';
316
- opts.remainingRetry;
317
- opts.remainingRedirects;
318
- opts.agent = otherHttpAgent;
319
-
320
- // READ-ONLY options (not exhaustive)
321
- opts.url; // DOT NOT OVERWRITE
322
- opts.maxRetry;
323
- opts.maxRedirects;
324
- opts.prevError; // error of previous request on retry
325
- opts.prevStatusCode; // HTTP status code of previous request on retry/redirect
326
-
327
- // opts must be returned
328
- return opts;
329
- },
330
- }
331
- ```
332
-
333
- Create a new instance with specific parameters instead of modifying global `rock.defaults`.
334
-
335
- By default, this new instance inherits values of the instance source if options are not overwritten.
336
- Headers are merged. Then only the first level of the options object is merged (no deep travelling in sub-objects or arrays).
337
-
338
- The `keepAliveDuration` can be changed only with `extend` method because `rock-req` creates new http Agent on new instances.
339
-
340
- ```js
341
- const myInstance = rock.extend({
342
- keepAliveDuration : 0, // Change keep alive duration. Default to 3000ms. Set 0 to deactivate keep alive.
343
- headers: {
344
- 'Custom-header': 'x-for-proxy'
345
- },
346
- timeout : 1000
347
- });
348
-
349
- myInstance.get('http://example.com', function (err, res, data) {})
350
- ```
351
-
352
- ### Intercept retries for Higher Availability / Higher bandwidth
353
-
354
- `beforeRequest` is always called on each request, each redirect and each retry.
355
- - on redirect, `opts.url` (and `hostname`, `port`, `protocol`, `path`) is updated to the new location. `opts.url` is null if it is a relative redirect.
356
- - on retry, `opts.url` (and `hostname`, `port`, `protocol`, `path`) have the same value as they did
357
- when the rock-req was initially called.
358
-
359
- For example, you can dynamically change the http Agent to use a another proxy on each request.
360
- Be careful, in this case, you must provide the right http/https Agent if there is a redirection from http to https.
361
- Otherwise, rock-req automatically replaces your Agent with the correct one if the protocol changes after redirection.
362
-
363
- Or, you can rewrite the URL if you want to use [HAProxy as a forward proxy](https://www.haproxy.com/user-spotlight-series/haproxy-as-egress-controller/).
364
-
365
-
366
- ```js
367
- const myInstance = rock.extend({
368
- beforeRequest: (opts) => {
369
- const { hostname, port, protocol, path } = opts;
370
- opts.protocol = 'http:';
371
- opts.hostname = '10.0.0.1';
372
- opts.port = 80;
373
- opts.path = `${hostname}/${port}${path}`;
374
- return opts;
375
- }
376
- });
377
-
378
- myInstance.get('http://example.com', function (err, res, data) {})
379
-
380
- ```
381
-
382
-
383
- ### Timeout
384
-
385
- You can set a timeout (in milliseconds) on the request with the `timeout` option.
386
- If the request takes longer than `timeout` to complete, then the entire request
387
- will fail with an `Error`.
388
-
389
- ```js
390
- const rock = require('rock-req')
391
-
392
- const opts = {
393
- url: 'http://example.com',
394
- timeout: 2000 // 2 second timeout
395
- }
396
-
397
- rock(opts, function (err, res, data) {})
398
- ```
399
-
400
-
401
- ### One Quick Tip
402
-
403
- It's a good idea to set the `'user-agent'` header so the provider can more easily
404
- see how their resource is used.
405
-
406
- ```js
407
- const rock = require('rock-req')
408
- const pkg = require('./package.json')
409
-
410
- rock({
411
- url : 'http://example.com',
412
- headers: {
413
- 'user-agent': `my-module/${pkg.version} (https://github.com/username/my-module)`
414
- }
415
- }, function (err, res, data) {})
416
- ```
417
-
418
- ### Proxies
419
-
420
- You can use the [`tunnel`](https://github.com/koichik/node-tunnel) module with the
421
- `agent` option to work with proxies:
422
-
423
- ```js
424
- const rock = require('rock-req')
425
- const tunnel = require('tunnel')
426
-
427
- const opts = {
428
- url: 'http://example.com',
429
- agent: tunnel.httpOverHttp({
430
- proxy: {
431
- host: 'localhost'
432
- }
433
- })
434
- }
435
-
436
- rock(opts, function (err, res, data) {})
437
- ```
438
-
439
- ### Cookies
440
-
441
- You can use the [`cookie`](https://github.com/jshttp/cookie) module to include
442
- cookies in a request:
443
-
444
- You can extend and create a new instance of rock-req to keep the cookie in header for each request.
445
-
446
- ```js
447
- const rock = require('rock-req')
448
- const cookie = require('cookie')
449
-
450
- const opts = {
451
- url: 'http://example.com',
452
- headers: {
453
- cookie: cookie.serialize('foo', 'bar')
454
- }
455
- }
456
-
457
- rock(opts, function (err, res, data) {})
458
- ```
459
-
460
- ### Form data
461
-
462
- You can use the [`form-data`](https://github.com/form-data/form-data) module to
463
- create POST request with form data:
464
-
465
- ```js
466
- const fs = require('fs')
467
- const rock = require('rock-req')
468
- const FormData = require('form-data')
469
- const form = new FormData()
470
-
471
- const opts = {
472
- url: 'http://example.com',
473
- body: () => {
474
- form.append('my_file', fs.createReadStream('/foo/bar.jpg'))
475
- }
476
- }
477
-
478
- rock.post(opts, function (err, res, data) {})
479
- ```
480
-
481
- #### Or, include `application/x-www-form-urlencoded` form data manually:
482
-
483
- ```js
484
- const rock = require('rock-req')
485
-
486
- const opts = {
487
- url: 'http://example.com',
488
- form: {
489
- key: 'value'
490
- }
491
- }
492
- rock.post(opts, function (err, res, data) {})
493
- ```
494
-
495
- ### Specifically disallowing redirects
496
-
497
- ```js
498
- const rock = require('rock-req')
499
-
500
- const opts = {
501
- url: 'http://example.com/will-redirect-elsewhere',
502
- followRedirects: false
503
- }
504
- // res.statusCode will be 301, no error thrown
505
- rock(opts, function (err, res, data) {})
506
- ```
507
-
508
- ### Basic Auth
509
-
510
- ```js
511
- const user = 'someuser'
512
- const pass = 'pa$$word'
513
- const encodedAuth = Buffer.from(`${user}:${pass}`).toString('base64')
514
-
515
- rock('http://example.com', {
516
- headers: {
517
- authorization: `Basic ${encodedAuth}`
518
- }
519
- })
520
- ```
521
-
522
- ### OAuth
523
-
524
- You can use the [`oauth-1.0a`](https://github.com/ddo/oauth-1.0a) module to create
525
- a signed OAuth request:
526
-
527
- ```js
528
- const rock = require('rock-req')
529
- const crypto = require('crypto')
530
- const OAuth = require('oauth-1.0a')
531
-
532
- const oauth = OAuth({
533
- consumer: {
534
- key: process.env.CONSUMER_KEY,
535
- secret: process.env.CONSUMER_SECRET
536
- },
537
- signature_method: 'HMAC-SHA1',
538
- hash_function: (baseString, key) => crypto.createHmac('sha1', key).update(baseString).digest('base64')
539
- })
540
-
541
- const token = {
542
- key: process.env.ACCESS_TOKEN,
543
- secret: process.env.ACCESS_TOKEN_SECRET
544
- }
545
-
546
- const url = 'https://api.twitter.com/1.1/statuses/home_timeline.json'
547
-
548
- const opts = {
549
- url: url,
550
- headers: oauth.toHeader(oauth.authorize({url, method: 'GET'}, token)),
551
- json: true
552
- }
553
-
554
- rock(opts, function (err, res) {})
555
- ```
556
-
557
- ### Throttle requests
558
-
559
- You can use [limiter](https://github.com/jhurliman/node-rate-limiter) to throttle requests. This is useful when calling an API that is rate limited.
560
-
561
- ```js
562
- const rock = require('rock-get')
563
- const RateLimiter = require('limiter').RateLimiter
564
- const limiter = new RateLimiter(1, 'second')
565
-
566
- const rock = (opts, cb) => limiter.removeTokens(1, () => rock(opts, cb))
567
- rock.concat = (opts, cb) => limiter.removeTokens(1, () => rock.concat(opts, cb))
568
-
569
- var opts = {
570
- url: 'http://example.com'
571
- }
572
-
573
- rock.concat(opts, processResult)
574
- rock.concat(opts, processResult)
575
-
576
- function processResult (err, res, data) {
577
- if (err) throw err
578
- console.log(data.toString())
579
- }
580
- ```
581
-
85
+ ## Documentation
582
86
 
583
- ## Breaking change for people coming from simplet-get
87
+ [The full documentation is here](doc/api.md) to reduce Node package file size.
584
88
 
585
- Rock-req is a fork of [simple-get](https://github.com/feross/simple-get)
586
-
587
- - Rock-req concatenates chunks and returns the concatenated buffer/JSON in the third paramater of the callback.
588
- Why? because 80% of the time, we need to do simple request without streams.
589
- When you need to pipe the result to another stream, you must use `opts.output` parameter (see "Output Stream" in the doc)
590
- - All streams must be created in a function. Rock-req return an error if it is not the case
591
- - before: `body = stream`
592
- - after `body = () => { const myStream = create(); return myStream; }`
593
-
594
-
595
- ## TODO:
596
-
597
- - [ ] replace deprecated `url.parse` by `new URL` but new URL is slower than url.parse. Let's see if Node 20 LTS is faster
598
- - [ ] promisify ?
599
- - [ ] typescript type ?
600
-
601
- # Supporters
89
+ ## Supporters
602
90
 
603
91
  This packaged in maintained by Carbone:
604
92
 
@@ -610,6 +98,3 @@ This packaged in maintained by Carbone:
610
98
 
611
99
 
612
100
  Thank you [Feross Aboukhadijeh](https://github.com/feross), creator of `simple-get`
613
-
614
-
615
-
package/index.js CHANGED
@@ -23,7 +23,7 @@ function extend (defaultOptions = {}) {
23
23
  }
24
24
  defaultOptions.headers = applyDefault(cloneLowerCase(defaultOptions.headers), _default.headers)
25
25
  _default = applyDefault(defaultOptions, _default) // inherits of parent options
26
- const agents = [http, https].map(h => (_default.keepAliveDuration > 0) ? new h.Agent({ keepAlive: true, keepAliveMsecs: _default.keepAliveDuration, timeout: _default.keepAliveDuration /* remove free socket node < 19 */ }) : undefined)
26
+ const agents = [http, https].map(h => (_default.keepAliveDuration > 0) ? new h.Agent({ keepAlive: true, keepAliveMsecs: _default.keepAliveDuration }) : undefined)
27
27
 
28
28
  function rock (opts, directBody, cb) {
29
29
  if (typeof opts === 'string') opts = { url: opts }
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "rock-req",
3
3
  "description": "Ultra-light (150 LOC, No dependencies) & Ultra-fast request library with reliable retry on failure, http/https, redirects, gzip/deflate/brotli, extensible, proxy, streams, JSON mode, forms, timeout",
4
- "version": "5.1.2",
4
+ "version": "5.1.3",
5
5
  "author": {
6
6
  "name": "David Grelaud & Feross Aboukhadijeh"
7
7
  },