loadtest 6.4.0 → 7.1.0

package/README.md

@@ -18,17 +18,9 @@ On Ubuntu or Mac OS X systems install using sudo:
 
     $ sudo npm install -g loadtest
 
-For access to the API just add package `loadtest` to your `package.json` devDependencies:
+For access to the API just install it in your `npm` package as a dev dependency:
 
-```json
-{
-	...
-	"devDependencies": {
-		"loadtest": "*"
-	},
-	...
-}
-```
+    $ npm install --save-dev loadtest
 
 ### Compatibility
 
@@ -36,8 +28,9 @@ Versions 6 and later should be used at least with Node.js v16 or later:
 
 * Node.js v16 or later: ^6.0.0
 * Node.js v10 or later: ^5.0.0
-* Node.js v8 or later: 4.x.y.
-* Node.js v6 or earlier: ^3.1.0.
+* Node.js v8 or later: 4.x.y
+* Node.js v6 or earlier: ^3.1.0
+* ES5 support (no `let`, `const` or arrow functions): ^2.0.0.
 
 ## Usage
 
@@ -88,24 +81,32 @@ so that you can abort deployment e.g. if 99% of the requests don't finish in 10
 
 ### Usage Don'ts
 
-`loadtest` saturates a single CPU pretty quickly.
-Do not use `loadtest` in this mode
-if the Node.js process is above 100% usage in `top`, which happens approx. when your load is above 1000~4000 rps.
+`loadtest` performance has improved significantly,
+but it is still limited.
+`loadtest` saturates a single CPU pretty quickly,
+so it uses half the available cores in your processor.
+If you see that the Node.js processes are above 100% usage in `top`,
+which happens approx. when your load is above 4000~5000 rps per core,
+please adjust the number of cores.
+So for instance with eight cores you can expect to get a maximum performance of
+8 * 5000 ~ 40 krps.
 (You can measure the practical limits of `loadtest` on your specific test machines by running it against a simple
 [test server](#test-server)
 and seeing when it reaches 100% CPU.)
-In this case try using in multi-process mode using the `--cores` parameter,
-see below.
 
-There are better tools for that use case:
+If you have reached the limits of `loadtest` even after using all cores,
+there are other tools that you can try.
 
+* [AutoCannon](https://www.npmjs.com/package/autocannon): also an `npm` package,
+awesome tool with an interface similar to `wrk`.
 * [Apache `ab`](http://httpd.apache.org/docs/2.2/programs/ab.html)
 has great performance, but it is also limited by a single CPU performance.
 Its practical limit is somewhere around ~40 krps.
 * [weighttp](http://redmine.lighttpd.net/projects/weighttp/wiki) is also `ab`-compatible
 and is supposed to be very fast (the author has not personally used it).
-* [wrk](https://github.com/wg/wrk) is multithreaded and fit for use when multiple CPUs are required or available.
+* [wrk](https://github.com/wg/wrk) is multithreaded and highly performance.
 It may need installing from source though, and its interface is not `ab`-compatible.
+* [wrk2](https://github.com/giltene/wrk2): evolution of `wrk`.
 
 ### Regular Usage
 
@@ -239,12 +240,12 @@ to provide the body of each request.
 This is useful if you want to generate request bodies dynamically and vary them for each request.
 For examples see above for `-p`.
 
-##### `-r`
+##### `-r recover`
 
 Recover from errors. Always active: loadtest does not stop on errors.
 After the tests are finished, if there were errors a report with all error codes will be shown.
 
-#### `-s`
+#### `-s secureProtocol`
 
 The TLS/SSL method to use. (e.g. TLSv1_method)
 
@@ -252,7 +253,7 @@ Example:
 
     $ loadtest -n 1000 -s TLSv1_method https://www.example.com
 
-#### `-V`
+#### `-V version`
 
 Show version number and exit.
 
@@ -283,15 +284,19 @@ Note: --rps is not supported for websockets.
 #### `--cores number`
 
 Start `loadtest` in multi-process mode on a number of cores simultaneously.
-Useful when a single CPU is saturated.
 Forks the requested number of processes using the
 [Node.js cluster module](https://nodejs.org/api/cluster.html).
+Default: half the available CPUs on the machine.
 
-In this mode the total number of requests and the rps rate are shared among all processes.
-The result returned is the aggregation of results from all cores.
+The total number of requests and the rps rate are shared among all processes.
+The result shown is the aggregation of results from all cores.
 
 Note: this option is not available in the API,
-where it runs just in the provided process.
+since it runs just within the calling process.
+
+**Warning**: the default value for `--cores` has changed in version 7+,
+from 1 to half the available CPUs on the machine.
+Set to 1 to get the previous single-process mode.
 
 #### `--timeout milliseconds`
 
@@ -496,351 +501,51 @@ However, it you try to push it beyond that, at 3 krps it will fail miserably.
 
 `loadtest` is not limited to running from the command line; it can be controlled using an API,
 thus allowing you to load test your application in your own tests.
+A short introduction follows; see [complete docs for API](doc/api.md).
 
 ### Invoke Load Test
 
-To run a load test, just `await` for the exported function `loadTest()` with the desired options, described below:
+To run a load test, invoke the exported function `loadTest()` with the desired options:
 
 ```javascript
 import {loadTest} from 'loadtest'
 
 const options = {
-	url: 'http://localhost:8000',
-	maxRequests: 1000,
+    url: 'http://localhost:8000',
+    maxRequests: 1000,
 }
 const result = await loadTest(options)
 result.show()
 console.log('Tests run successfully')
 ```
 
-The call returns a `Result` object that contains all info about the load test, also described below.
-Call `result.show()` to display the results in the standard format on the console.
-
-As a legacy from before promises existed,
-if an optional callback is passed as second parameter then it will not behave as `async`:
-the callback `function(error, result)` will be invoked when the max number of requests is reached,
-or when the max number of seconds has elapsed.
-
-```javascript
-import {loadTest} from 'loadtest'
-
-const options = {
-	url: 'http://localhost:8000',
-	maxRequests: 1000,
-}
-loadTest(options, function(error, result) {
-	if (error) {
-		return console.error('Got an error: %s', error)
-	}
-	result.show()
-	console.log('Tests run successfully')
-})
-```
-
-
-Beware: if there are no `maxRequests` and no `maxSeconds`, then tests will run forever
-and will not call the callback.
-
-### Result
-
-The latency result returned at the end of the load test contains a full set of data, including:
-mean latency, number of errors and percentiles.
-A simplified example follows:
-
-```javascript
-{
-  url: 'http://localhost:80/',
-  maxRequests: 1000,
-  maxSeconds: 0,
-  concurrency: 10,
-  agent: 'none',
-  requestsPerSecond: undefined,
-  totalRequests: 1000,
-  percentiles: {
-	'50': 7,
-	'90': 10,
-	'95': 11,
-	'99': 15
-  },
-  effectiveRps: 2824,
-  elapsedSeconds: 0.354108,
-  meanLatencyMs: 7.72,
-  maxLatencyMs: 20,
-  totalErrors: 3,
-  errorCodes: {
-	'0': 1,
-	'500': 2
-  },
-}
-```
-
-The `result` object also has a `result.show()` function
-that displays the results on the console in the standard format.
-
-### Options
-
-All options but `url` are, as their name implies, optional.
-
-#### `url`
-
-The URL to invoke. Mandatory.
-
-#### `concurrency`
-
-How many clients to start in parallel.
-
-#### `maxRequests`
-
-A max number of requests; after they are reached the test will end.
-
-Note: the actual number of requests sent can be bigger if there is a concurrency level;
-loadtest will report just on the max number of requests.
-
-#### `maxSeconds`
-
-Max number of seconds to run the tests.
-
-Note: after the given number of seconds `loadtest` will stop sending requests,
-but may continue receiving tests afterwards.
-
-#### `timeout`
-
-Timeout for each generated request in milliseconds. Setting this to 0 disables timeout (default).
-
-#### `cookies`
-
-An array of cookies to send. Each cookie should be a string of the form name=value.
-
-#### `headers`
-
-A map of headers. Each header should be an entry in the map with the value given as a string.
-If you want to have several values for a header, write a single value separated by semicolons,
-like this:
-
-    {
-        accept: "text/plain;text/html"
-    }
-
-Note: when using the API, the "host" header is not inferred from the URL but needs to be sent
-explicitly.
-
-#### `method`
-
-The method to use: POST, PUT. Default: GET.
-
-#### `body`
-
-The contents to send in the body of the message, for POST or PUT requests.
-Can be a string or an object (which will be converted to JSON).
-
-#### `contentType`
-
-The MIME type to use for the body. Default content type is `text/plain`.
-
-#### `requestsPerSecond`
-
-How many requests each client will send per second.
-
-#### `requestGenerator`
-
-Use a custom request generator function.
-The request needs to be generated synchronously and returned when this function is invoked.
-
-Example request generator function could look like this:
-
-```javascript
-function(params, options, client, callback) {
-  const message = generateMessage();
-  const request = client(options, callback);
-  options.headers['Content-Length'] = message.length;
-  options.headers['Content-Type'] = 'application/x-www-form-urlencoded';
-  request.write(message);
-  request.end();
-  return request;
-}
-```
-
-See [`sample/request-generator.js`](sample/request-generator.js) for some sample code including a body
-(or [`sample/request-generator.ts`](sample/request-generator.ts) for ES6/TypeScript).
-
-#### `agentKeepAlive`
-
-Use an agent with 'Connection: Keep-alive'.
-
-Note: Uses [agentkeepalive](https://npmjs.org/package/agentkeepalive),
-which performs better than the default node.js agent.
-
-#### `quiet` (deprecated)
-
-Do not show any messages.
-
-Note: deprecated in version 6+, shows a warning.
-
-#### `indexParam`
-
-The given string will be replaced in the final URL with a unique index.
-E.g.: if URL is `http://test.com/value` and `indexParam=value`, then the URL
-will be:
-
-* http://test.com/1
-* http://test.com/2
-* ...
-* body will also be replaced `body:{ userid: id_value }` will be `body:{ userid: id_1 }`
-
-#### `indexParamCallback`
-
-A function that would be executed to replace the value identified through `indexParam` through a custom value generator.
-
-E.g.: if URL is `http://test.com/value` and `indexParam=value` and
-```javascript
-indexParamCallback: function customCallBack() {
-  return Math.floor(Math.random() * 10); //returns a random integer from 0 to 9
-}
-``` 
-then the URL could be:
-
-* http://test.com/1 (Randomly generated integer 1)
-* http://test.com/5 (Randomly generated integer 5)
-* http://test.com/6 (Randomly generated integer 6)
-* http://test.com/8 (Randomly generated integer 8)
-* ...
-* body will also be replaced `body:{ userid: id_value }` will be `body:{ userid: id_<value from callback> }`
-
-#### `insecure`
-
-Allow invalid and self-signed certificates over https.
-
-#### `secureProtocol`
-
-The TLS/SSL method to use. (e.g. TLSv1_method)
-
-Example:
-
-```javascript
-import {loadTest} from 'loadtest'
-
-const options = {
-	url: 'https://www.example.com',
-    maxRequests: 100,
-    secureProtocol: 'TLSv1_method'
-}
-
-loadTest(options, function(error) {
-	if (error) {
-		return console.error('Got an error: %s', error)
-	}
-	console.log('Tests run successfully')
-})
-```
-
-#### `statusCallback`
-
-If present, this function executes after every request operation completes. Provides immediate access to the test result while the
-test batch is still running. This can be used for more detailed custom logging or developing your own spreadsheet or
-statistical analysis of the result.
-
-The result and error passed to the callback are in the same format as the result passed to the final callback.
- 
-In addition, the following three properties are added to the `result` object:
-
-- `requestElapsed`: time in milliseconds it took to complete this individual request.
-- `requestIndex`: 0-based index of this particular request in the sequence of all requests to be made.
-- `instanceIndex`: the `loadtest(...)` instance index. This is useful if you call `loadtest()` more than once.
-
-You will need to check if `error` is populated in order to determine which object to check for these properties.
-
-The second parameter contains info about the current request:
-
-```javascript
-{
-	host: 'localhost',
-	path: '/',
-	method: 'GET',
-	statusCode: 200,
-	body: '<html><body>hi</body></html>',
-	headers: [...]
-}
-```
-
-Example:
-
-```javascript
-import {loadTest} from 'loadtest'
-
-function statusCallback(error, result, latency) {
-    console.log('Current latency %j, result %j, error %j', latency, result, error)
-    console.log('----')
-    console.log('Request elapsed milliseconds: ', result.requestElapsed)
-    console.log('Request index: ', result.requestIndex)
-    console.log('Request loadtest() instance index: ', result.instanceIndex)
-}
-
-const options = {
-    url: 'http://localhost:8000',
-    maxRequests: 1000,
-    statusCallback: statusCallback
-}
-
-loadTest(options, function(error) {
-    if (error) {
-        return console.error('Got an error: %s', error)
-    }
-    console.log('Tests run successfully')
-})
-```
- 
-In some situations request data needs to be available in the statusCallBack.
-This data can be assigned to `request.labels` in the requestGenerator:
-```javascript
-const options = {
-	// ...
-	requestGenerator: (params, options, client, callback) => {
-		// ...
-        const randomInputData = Math.random().toString().substr(2, 8);
-        const message = JSON.stringify({ randomInputData })
-		const request = client(options, callback);
-        request.labels = randomInputData;
-		request.write(message);
-		return request;
-	}
-};
-```
-
-Then in statusCallBack the labels can be accessed through `result.labels`:
-```javascript
-function statusCallback(error, result, latency) {
-    console.log(result.labels);
-}
-```
-
-**Warning**: The format for `statusCallback` has changed in version 2.0.0 onwards.
-It used to be `statusCallback(latency, result, error)`,
-it has been changed to conform to the usual Node.js standard.
-
-#### `contentInspector`
-
-A function that would be executed after every request before its status be added to the final statistics.
-
-The is can be used when you want to mark some result with 200 http status code to be failed or error.
-
-The `result` object passed to this callback function has the same fields as the `result` object passed to `statusCallback`.
-
-`customError` can be added to mark this result as failed or error. `customErrorCode` will be provided in the final statistics, in addtion to the http status code.
-
-Example:
-
-```javascript
-function contentInspector(result) {
-    if (result.statusCode == 200) {
-        const body = JSON.parse(result.body)
-        // how to examine the body depends on the content that the service returns
-        if (body.status.err_code !== 0) {
-            result.customError = body.status.err_code + " " + body.status.msg
-            result.customErrorCode = body.status.err_code
-        }
-    }
-},
-```
+Beware: if there are no `maxRequests` and no `maxSeconds`, the test will run forever.
+
+### `loadTest()` Parameters
+
+A simplified list of parameters is shown below;
+see [doc/api.md](doc/api.md) for the full explanations with examples.
+
+* `url`: URL to invoke, mandatory.
+* `concurrency`: how many clients to start in parallel.
+* `maxRequests`: max number of requests; after they are reached the test will end.
+* `maxSeconds`: max number of seconds to run the tests.
+* `timeout`: timeout for each generated request in milliseconds, set to 0 to disable (default).
+* `cookies`: array of cookies to send, of the form `name=value`.
+* `headers`: object with headers, each with the value as string. Separate by semicolons to have multiple values.
+* `method`: HTTP method to use, default `GET`.
+* `body`: contents to send in the body of the message.
+* `contentType`: MIME type to use for the body, default `text/plain`.
+* `requestsPerSecond`: how many requests will be sent per second.
+* `requestGenerator`: custom request generator function.
+* `agentKeepAlive`: if true, will use 'Connection: Keep-alive'.
+* `quiet`: if true, do not show any messages.
+* `indexParam`: parameter to replace in URL and body with a unique index.
+* `indexParamCallback`: function to generate unique indexes.
+* `insecure`: allow invalid and self-signed certificates over https.
+* `secureProtocol`: TLS/SSL method to use.
+* `statusCallback(error, result)`: function to call after every request is completed.
+* `contentInspector(result)`: function to call before aggregating statistics.
     
 ### Start Test Server
 
@@ -853,40 +558,14 @@ const server = await startServer({port: 8000})
 await server.close()
 ```
 
-This function returns when the server is up and running,
-with an HTTP server which can be `close()`d when it is no longer useful.
-As a legacy from before promises existed,
-if an optional callback is passed as second parameter then it will not behave as `async`:
-
-```
-const server = startServer({port: 8000}, error => console.error(error))
-```
-
-The following options are available.
-
-#### `port`
-
-Optional port to use for the server.
-
-Note: the default port is 7357, since port 80 requires special privileges.
+The following options are available,
+see [doc/api.md](doc/api.md) for details.
 
-#### `delay`
-
-Wait the given number of milliseconds to answer each request.
-
-#### `error`
-
-Return an HTTP error code.
-
-#### `percent`
-
-Return an HTTP error code only for the given % of requests.
-If no error code was specified, default is 500.
-
-#### `logger`
-
-A function to be called as `logger(request, response)` after every request served by the test server.
-Where `request` and `response` are the usual HTTP objects.
+* `port`: optional port to use for the server, default 7357.
+* `delay`: milliseconds to wait before answering each request.
+* `error`: HTTP status code to return, default 200 (no error).
+* `percent`: return error only for the given % of requests.
+* `logger(request, response)`: function to call after every request.
 
 ### Configuration file
 
@@ -932,14 +611,8 @@ For more information about the actual configuration file name, read the [confino
 
 ### Complete Example
 
-The file `test/integration.js` shows a complete example, which is also a full integration test:
-it starts the server, send 1000 requests, waits for the callback and closes down the server.
-
-## Versioning
-
-Version 3.x uses ES2015 (ES6) features,
-such as `const` or `let` and arrow functions.
-For ES5 support please use versions 2.x.
+The file `test/integration.js` contains complete examples, which are also a full integration test suite:
+they start the server with different options, send requests, waits for finalization and close down the server.
 
 ## Licensed under The MIT License
 

package/bin/loadtest.js

@@ -5,6 +5,7 @@ import * as stdio from 'stdio'
 import {loadTest} from '../lib/loadtest.js'
 import {runTask} from '../lib/cluster.js'
 import {Result} from '../lib/result.js'
+import {getHalfCores} from '../lib/cluster.js'
 
 
 const options = stdio.getopt({
@@ -34,7 +35,7 @@ const options = stdio.getopt({
 	key: {args: 1, description: 'The client key to use'},
 	cert: {args: 1, description: 'The client certificate to use'},
 	quiet: {description: 'Do not log any messages'},
-	cores: {args: 1, description: 'Number of cores to use', default: 1},
+	cores: {args: 1, description: 'Number of cores to use', default: getHalfCores()},
 	agent: {description: 'Use a keep-alive http agent (deprecated)'},
 	debug: {description: 'Show debug messages (deprecated)'},
 });