loadtest 6.3.2 → 7.0.0

package/README.md

@@ -18,17 +18,9 @@ On Ubuntu or Mac OS X systems install using sudo:
 
     $ sudo npm install -g loadtest
 
-For access to the API just add package `loadtest` to your `package.json` devDependencies:
+For access to the API just install it in your `npm` package as a dev dependency:
 
-```json
-{
-	...
-	"devDependencies": {
-		"loadtest": "*"
-	},
-	...
-}
-```
+    $ npm install --save-dev loadtest
 
 ### Compatibility
 
@@ -88,24 +80,32 @@ so that you can abort deployment e.g. if 99% of the requests don't finish in 10
 
 ### Usage Don'ts
 
-`loadtest` saturates a single CPU pretty quickly.
-Do not use `loadtest` in this mode
-if the Node.js process is above 100% usage in `top`, which happens approx. when your load is above 1000~4000 rps.
+`loadtest` performance has improved significantly,
+but it is still limited.
+`loadtest` saturates a single CPU pretty quickly,
+so it uses half the available cores in your processor.
+If you see that the Node.js processes are above 100% usage in `top`,
+which happens approx. when your load is above 4000~5000 rps per core,
+please adjust the number of cores.
+So for instance with eight cores you can expect to get a maximum performance of
+8 * 5000 ~ 40 krps.
 (You can measure the practical limits of `loadtest` on your specific test machines by running it against a simple
 [test server](#test-server)
 and seeing when it reaches 100% CPU.)
-In this case try using in multi-process mode using the `--cores` parameter,
-see below.
 
-There are better tools for that use case:
+If you have reached the limits of `loadtest` even after using all cores,
+there are other tools that you can try.
 
+* [AutoCannon](https://www.npmjs.com/package/autocannon): also an `npm` package,
+awesome tool with an interface similar to `wrk`.
 * [Apache `ab`](http://httpd.apache.org/docs/2.2/programs/ab.html)
 has great performance, but it is also limited by a single CPU performance.
 Its practical limit is somewhere around ~40 krps.
 * [weighttp](http://redmine.lighttpd.net/projects/weighttp/wiki) is also `ab`-compatible
 and is supposed to be very fast (the author has not personally used it).
-* [wrk](https://github.com/wg/wrk) is multithreaded and fit for use when multiple CPUs are required or available.
+* [wrk](https://github.com/wg/wrk) is multithreaded and highly performance.
 It may need installing from source though, and its interface is not `ab`-compatible.
+* [wrk2](https://github.com/giltene/wrk2): evolution of `wrk`.
 
 ### Regular Usage
 
@@ -239,12 +239,12 @@ to provide the body of each request.
 This is useful if you want to generate request bodies dynamically and vary them for each request.
 For examples see above for `-p`.
 
-##### `-r`
+##### `-r recover`
 
 Recover from errors. Always active: loadtest does not stop on errors.
 After the tests are finished, if there were errors a report with all error codes will be shown.
 
-#### `-s`
+#### `-s secureProtocol`
 
 The TLS/SSL method to use. (e.g. TLSv1_method)
 
@@ -252,7 +252,7 @@ Example:
 
     $ loadtest -n 1000 -s TLSv1_method https://www.example.com
 
-#### `-V`
+#### `-V version`
 
 Show version number and exit.
 
@@ -283,15 +283,19 @@ Note: --rps is not supported for websockets.
 #### `--cores number`
 
 Start `loadtest` in multi-process mode on a number of cores simultaneously.
-Useful when a single CPU is saturated.
 Forks the requested number of processes using the
 [Node.js cluster module](https://nodejs.org/api/cluster.html).
+Default: half the available CPUs on the machine.
 
-In this mode the total number of requests and the rps rate are shared among all processes.
-The result returned is the aggregation of results from all cores.
+The total number of requests and the rps rate are shared among all processes.
+The result shown is the aggregation of results from all cores.
 
 Note: this option is not available in the API,
-where it runs just in the provided process.
+since it runs just within the calling process.
+
+**Warning**: the default value for `--cores` has changed in version 7+,
+from 1 to half the available CPUs on the machine.
+Set to 1 to get the previous single-process mode.
 
 #### `--timeout milliseconds`
 
@@ -496,352 +500,51 @@ However, it you try to push it beyond that, at 3 krps it will fail miserably.
 
 `loadtest` is not limited to running from the command line; it can be controlled using an API,
 thus allowing you to load test your application in your own tests.
+A short introduction follows; see [complete docs for API](doc/api.md).
 
 ### Invoke Load Test
 
-To run a load test, just `await` for the exported function `loadTest()` with the desired options, described below:
+To run a load test, invoke the exported function `loadTest()` with the desired options:
 
 ```javascript
 import {loadTest} from 'loadtest'
 
 const options = {
-	url: 'http://localhost:8000',
-	maxRequests: 1000,
+    url: 'http://localhost:8000',
+    maxRequests: 1000,
 }
 const result = await loadTest(options)
 result.show()
 console.log('Tests run successfully')
-})
-```
-
-The call returns a `Result` object that contains all info about the load test, also described below.
-Call `result.show()` to display the results in the standard format on the console.
-
-As a legacy from before promises existed,
-if an optional callback is passed as second parameter then it will not behave as `async`:
-the callback `function(error, result)` will be invoked when the max number of requests is reached,
-or when the max number of seconds has elapsed.
-
-```javascript
-import {loadTest} from 'loadtest'
-
-const options = {
-	url: 'http://localhost:8000',
-	maxRequests: 1000,
-}
-loadTest(options, function(error, result) {
-	if (error) {
-		return console.error('Got an error: %s', error)
-	}
-	result.show()
-	console.log('Tests run successfully')
-})
-```
-
-
-Beware: if there are no `maxRequests` and no `maxSeconds`, then tests will run forever
-and will not call the callback.
-
-### Result
-
-The latency result returned at the end of the load test contains a full set of data, including:
-mean latency, number of errors and percentiles.
-A simplified example follows:
-
-```javascript
-{
-  url: 'http://localhost:80/',
-  maxRequests: 1000,
-  maxSeconds: 0,
-  concurrency: 10,
-  agent: 'none',
-  requestsPerSecond: undefined,
-  totalRequests: 1000,
-  percentiles: {
-	'50': 7,
-	'90': 10,
-	'95': 11,
-	'99': 15
-  },
-  effectiveRps: 2824,
-  elapsedSeconds: 0.354108,
-  meanLatencyMs: 7.72,
-  maxLatencyMs: 20,
-  totalErrors: 3,
-  errorCodes: {
-	'0': 1,
-	'500': 2
-  },
-}
-```
-
-The `result` object also has a `result.show()` function
-that displays the results on the console in the standard format.
-
-### Options
-
-All options but `url` are, as their name implies, optional.
-
-#### `url`
-
-The URL to invoke. Mandatory.
-
-#### `concurrency`
-
-How many clients to start in parallel.
-
-#### `maxRequests`
-
-A max number of requests; after they are reached the test will end.
-
-Note: the actual number of requests sent can be bigger if there is a concurrency level;
-loadtest will report just on the max number of requests.
-
-#### `maxSeconds`
-
-Max number of seconds to run the tests.
-
-Note: after the given number of seconds `loadtest` will stop sending requests,
-but may continue receiving tests afterwards.
-
-#### `timeout`
-
-Timeout for each generated request in milliseconds. Setting this to 0 disables timeout (default).
-
-#### `cookies`
-
-An array of cookies to send. Each cookie should be a string of the form name=value.
-
-#### `headers`
-
-A map of headers. Each header should be an entry in the map with the value given as a string.
-If you want to have several values for a header, write a single value separated by semicolons,
-like this:
-
-    {
-        accept: "text/plain;text/html"
-    }
-
-Note: when using the API, the "host" header is not inferred from the URL but needs to be sent
-explicitly.
-
-#### `method`
-
-The method to use: POST, PUT. Default: GET.
-
-#### `body`
-
-The contents to send in the body of the message, for POST or PUT requests.
-Can be a string or an object (which will be converted to JSON).
-
-#### `contentType`
-
-The MIME type to use for the body. Default content type is `text/plain`.
-
-#### `requestsPerSecond`
-
-How many requests each client will send per second.
-
-#### `requestGenerator`
-
-Use a custom request generator function.
-The request needs to be generated synchronously and returned when this function is invoked.
-
-Example request generator function could look like this:
-
-```javascript
-function(params, options, client, callback) {
-  const message = generateMessage();
-  const request = client(options, callback);
-  options.headers['Content-Length'] = message.length;
-  options.headers['Content-Type'] = 'application/x-www-form-urlencoded';
-  request.write(message);
-  request.end();
-  return request;
-}
-```
-
-See [`sample/request-generator.js`](sample/request-generator.js) for some sample code including a body
-(or [`sample/request-generator.ts`](sample/request-generator.ts) for ES6/TypeScript).
-
-#### `agentKeepAlive`
-
-Use an agent with 'Connection: Keep-alive'.
-
-Note: Uses [agentkeepalive](https://npmjs.org/package/agentkeepalive),
-which performs better than the default node.js agent.
-
-#### `quiet` (deprecated)
-
-Do not show any messages.
-
-Note: deprecated in version 6+, shows a warning.
-
-#### `indexParam`
-
-The given string will be replaced in the final URL with a unique index.
-E.g.: if URL is `http://test.com/value` and `indexParam=value`, then the URL
-will be:
-
-* http://test.com/1
-* http://test.com/2
-* ...
-* body will also be replaced `body:{ userid: id_value }` will be `body:{ userid: id_1 }`
-
-#### `indexParamCallback`
-
-A function that would be executed to replace the value identified through `indexParam` through a custom value generator.
-
-E.g.: if URL is `http://test.com/value` and `indexParam=value` and
-```javascript
-indexParamCallback: function customCallBack() {
-  return Math.floor(Math.random() * 10); //returns a random integer from 0 to 9
-}
-``` 
-then the URL could be:
-
-* http://test.com/1 (Randomly generated integer 1)
-* http://test.com/5 (Randomly generated integer 5)
-* http://test.com/6 (Randomly generated integer 6)
-* http://test.com/8 (Randomly generated integer 8)
-* ...
-* body will also be replaced `body:{ userid: id_value }` will be `body:{ userid: id_<value from callback> }`
-
-#### `insecure`
-
-Allow invalid and self-signed certificates over https.
-
-#### `secureProtocol`
-
-The TLS/SSL method to use. (e.g. TLSv1_method)
-
-Example:
-
-```javascript
-import {loadTest} from 'loadtest'
-
-const options = {
-	url: 'https://www.example.com',
-    maxRequests: 100,
-    secureProtocol: 'TLSv1_method'
-}
-
-loadTest(options, function(error) {
-	if (error) {
-		return console.error('Got an error: %s', error)
-	}
-	console.log('Tests run successfully')
-})
-```
-
-#### `statusCallback`
-
-If present, this function executes after every request operation completes. Provides immediate access to the test result while the
-test batch is still running. This can be used for more detailed custom logging or developing your own spreadsheet or
-statistical analysis of the result.
-
-The result and error passed to the callback are in the same format as the result passed to the final callback.
- 
-In addition, the following three properties are added to the `result` object:
-
-- `requestElapsed`: time in milliseconds it took to complete this individual request.
-- `requestIndex`: 0-based index of this particular request in the sequence of all requests to be made.
-- `instanceIndex`: the `loadtest(...)` instance index. This is useful if you call `loadtest()` more than once.
-
-You will need to check if `error` is populated in order to determine which object to check for these properties.
-
-The second parameter contains info about the current request:
-
-```javascript
-{
-	host: 'localhost',
-	path: '/',
-	method: 'GET',
-	statusCode: 200,
-	body: '<html><body>hi</body></html>',
-	headers: [...]
-}
-```
-
-Example:
-
-```javascript
-import {loadTest} from 'loadtest'
-
-function statusCallback(error, result, latency) {
-    console.log('Current latency %j, result %j, error %j', latency, result, error)
-    console.log('----')
-    console.log('Request elapsed milliseconds: ', result.requestElapsed)
-    console.log('Request index: ', result.requestIndex)
-    console.log('Request loadtest() instance index: ', result.instanceIndex)
-}
-
-const options = {
-    url: 'http://localhost:8000',
-    maxRequests: 1000,
-    statusCallback: statusCallback
-}
-
-loadTest(options, function(error) {
-    if (error) {
-        return console.error('Got an error: %s', error)
-    }
-    console.log('Tests run successfully')
-})
 ```
- 
-In some situations request data needs to be available in the statusCallBack.
-This data can be assigned to `request.labels` in the requestGenerator:
-```javascript
-const options = {
-	// ...
-	requestGenerator: (params, options, client, callback) => {
-		// ...
-        const randomInputData = Math.random().toString().substr(2, 8);
-        const message = JSON.stringify({ randomInputData })
-		const request = client(options, callback);
-        request.labels = randomInputData;
-		request.write(message);
-		return request;
-	}
-};
-```
-
-Then in statusCallBack the labels can be accessed through `result.labels`:
-```javascript
-function statusCallback(error, result, latency) {
-    console.log(result.labels);
-}
-```
-
-**Warning**: The format for `statusCallback` has changed in version 2.0.0 onwards.
-It used to be `statusCallback(latency, result, error)`,
-it has been changed to conform to the usual Node.js standard.
-
-#### `contentInspector`
-
-A function that would be executed after every request before its status be added to the final statistics.
 
-The is can be used when you want to mark some result with 200 http status code to be failed or error.
-
-The `result` object passed to this callback function has the same fields as the `result` object passed to `statusCallback`.
-
-`customError` can be added to mark this result as failed or error. `customErrorCode` will be provided in the final statistics, in addtion to the http status code.
-
-Example:
-
-```javascript
-function contentInspector(result) {
-    if (result.statusCode == 200) {
-        const body = JSON.parse(result.body)
-        // how to examine the body depends on the content that the service returns
-        if (body.status.err_code !== 0) {
-            result.customError = body.status.err_code + " " + body.status.msg
-            result.customErrorCode = body.status.err_code
-        }
-    }
-},
-```
+Beware: if there are no `maxRequests` and no `maxSeconds`, the test will run forever.
+
+### `loadTest()` Parameters
+
+A simplified list of parameters is shown below;
+see [doc/api.md](doc/api.md) for the full explanations with examples.
+
+* `url`: URL to invoke, mandatory.
+* `concurrency`: how many clients to start in parallel.
+* `maxRequests`: max number of requests; after they are reached the test will end.
+* `maxSeconds`: max number of seconds to run the tests.
+* `timeout`: timeout for each generated request in milliseconds, set to 0 to disable (default).
+* `cookies`: array of cookies to send, of the form `name=value`.
+* `headers`: object with headers, each with the value as string. Separate by semicolons to have multiple values.
+* `method`: HTTP method to use, default `GET`.
+* `body`: contents to send in the body of the message.
+* `contentType`: MIME type to use for the body, default `text/plain`.
+* `requestsPerSecond`: how many requests will be sent per second.
+* `requestGenerator`: custom request generator function.
+* `agentKeepAlive`: if true, will use 'Connection: Keep-alive'.
+* `quiet`: if true, do not show any messages.
+* `indexParam`: parameter to replace in URL and body with a unique index.
+* `indexParamCallback`: function to generate unique indexes.
+* `insecure`: allow invalid and self-signed certificates over https.
+* `secureProtocol`: TLS/SSL method to use.
+* `statusCallback(error, result)`: function to call after every request is completed.
+* `contentInspector(result)`: function to call before aggregating statistics.
     
 ### Start Test Server
 
@@ -854,35 +557,15 @@ const server = await startServer({port: 8000})
 await server.close()
 ```
 
-This function returns when the server is up and running,
-with an HTTP server which can be `close()`d when it is no longer useful.
-As a legacy from before promises existed,
-if an optional callback is passed as second parameter then it will not behave as `async`:
-
-```
-const server = startServer({port: 8000}, error => console.error(error))
-```
-
-The following options are available.
-
-#### `port`
-
-Optional port to use for the server.
-
-Note: the default port is 7357, since port 80 requires special privileges.
-
-#### `delay`
-
-Wait the given number of milliseconds to answer each request.
-
-#### `error`
-
-Return an HTTP error code.
-
-#### `percent`
+The following options are available:
+* `port`: optional port to use for the server, default 7357.
+* `delay`: milliseconds to wait before answering each request.
+* `error`: HTTP status code to return, default 200 (no error).
+* `percent`: return error only for the given % of requests.
+* `logger(request, response)`
 
-Return an HTTP error code only for the given % of requests.
-If no error code was specified, default is 500.
+A function to be called as `logger(request, response)` after every request served by the test server.
+Where `request` and `response` are the usual HTTP objects.
 
 ### Configuration file
 

package/bin/loadtest.js

@@ -5,6 +5,7 @@ import * as stdio from 'stdio'
 import {loadTest} from '../lib/loadtest.js'
 import {runTask} from '../lib/cluster.js'
 import {Result} from '../lib/result.js'
+import {getHalfCores} from '../lib/cluster.js'
 
 
 const options = stdio.getopt({
@@ -34,7 +35,7 @@ const options = stdio.getopt({
 	key: {args: 1, description: 'The client key to use'},
 	cert: {args: 1, description: 'The client certificate to use'},
 	quiet: {description: 'Do not log any messages'},
-	cores: {args: 1, description: 'Number of cores to use', default: 1},
+	cores: {args: 1, description: 'Number of cores to use', default: getHalfCores()},
 	agent: {description: 'Use a keep-alive http agent (deprecated)'},
 	debug: {description: 'Show debug messages (deprecated)'},
 });

package/doc/api.md

@@ -0,0 +1,366 @@
+# loadtest API
+
+The `loadtest` package is a powerful tool that can be added to your package.
+The API allows for easy integration in your own tests,
+for instance to run automated load tests.
+You can verify the performance of your server before each deployment,
+and deploy only if a certain target is reached.
+
+## Installation
+
+For access to the API just install it in your `npm` package as a dev dependency:
+
+    $ npm install --save-dev loadtest
+
+### Compatibility
+
+See [README file](../README.md) for compatibility.
+
+## API
+
+`loadtest` is not limited to running from the command line; it can be controlled using an API,
+thus allowing you to load test your application in your own tests.
+
+### Invoke Load Test
+
+To run a load test, just `await` for the exported function `loadTest()` with the desired options, described below:
+
+```javascript
+import {loadTest} from 'loadtest'
+
+const options = {
+	url: 'http://localhost:8000',
+	maxRequests: 1000,
+}
+const result = await loadTest(options)
+result.show()
+console.log('Tests run successfully')
+```
+
+The call returns a `Result` object that contains all info about the load test, also described below.
+Call `result.show()` to display the results in the standard format on the console.
+
+As a legacy from before promises existed,
+if an optional callback is passed as second parameter then it will not behave as `async`:
+the callback `function(error, result)` will be invoked when the max number of requests is reached,
+or when the max number of seconds has elapsed.
+
+```javascript
+import {loadTest} from 'loadtest'
+
+const options = {
+	url: 'http://localhost:8000',
+	maxRequests: 1000,
+}
+loadTest(options, function(error, result) {
+	if (error) {
+		return console.error('Got an error: %s', error)
+	}
+	result.show()
+	console.log('Tests run successfully')
+})
+```
+
+
+Beware: if there are no `maxRequests` and no `maxSeconds`, then tests will run forever
+and will not call the callback.
+
+### Result
+
+The latency result returned at the end of the load test contains a full set of data, including:
+mean latency, number of errors and percentiles.
+A simplified example follows:
+
+```javascript
+{
+  url: 'http://localhost:80/',
+  maxRequests: 1000,
+  maxSeconds: 0,
+  concurrency: 10,
+  agent: 'none',
+  requestsPerSecond: undefined,
+  totalRequests: 1000,
+  percentiles: {
+	'50': 7,
+	'90': 10,
+	'95': 11,
+	'99': 15
+  },
+  effectiveRps: 2824,
+  elapsedSeconds: 0.354108,
+  meanLatencyMs: 7.72,
+  maxLatencyMs: 20,
+  totalErrors: 3,
+  errorCodes: {
+	'0': 1,
+	'500': 2
+  },
+}
+```
+
+The `result` object also has a `result.show()` function
+that displays the results on the console in the standard format.
+
+### Options
+
+All options but `url` are, as their name implies, optional.
+See also the [simplified list](../README.md#loadtest-parameters).
+
+#### `url`
+
+The URL to invoke. Mandatory.
+
+#### `concurrency`
+
+How many clients to start in parallel.
+
+#### `maxRequests`
+
+A max number of requests; after they are reached the test will end.
+
+Note: the actual number of requests sent can be bigger if there is a concurrency level;
+loadtest will report just on the max number of requests.
+
+#### `maxSeconds`
+
+Max number of seconds to run the tests.
+
+Note: after the given number of seconds `loadtest` will stop sending requests,
+but may continue receiving tests afterwards.
+
+#### `timeout`
+
+Timeout for each generated request in milliseconds. Setting this to 0 disables timeout (default).
+
+#### `cookies`
+
+An array of cookies to send. Each cookie should be a string of the form name=value.
+
+#### `headers`
+
+A map of headers. Each header should be an entry in the map with the value given as a string.
+If you want to have several values for a header, write a single value separated by semicolons,
+like this:
+
+    {
+        accept: "text/plain;text/html"
+    }
+
+Note: when using the API, the "host" header is not inferred from the URL but needs to be sent
+explicitly.
+
+#### `method`
+
+The method to use: POST, PUT. Default: GET.
+
+#### `body`
+
+The contents to send in the body of the message, for POST or PUT requests.
+Can be a string or an object (which will be converted to JSON).
+
+#### `contentType`
+
+The MIME type to use for the body. Default content type is `text/plain`.
+
+#### `requestsPerSecond`
+
+How many requests will be sent per second globally.
+
+#### `requestGenerator(params, options, client, callback)`
+
+Use a custom request generator function.
+The request needs to be generated synchronously and returned when this function is invoked.
+
+Example request generator function could look like this:
+
+```javascript
+function(params, options, client, callback) {
+  const message = generateMessage();
+  const request = client(options, callback);
+  options.headers['Content-Length'] = message.length;
+  options.headers['Content-Type'] = 'application/x-www-form-urlencoded';
+  request.write(message);
+  request.end();
+  return request;
+}
+```
+
+See [`sample/request-generator.js`](sample/request-generator.js) for some sample code including a body
+(or [`sample/request-generator.ts`](sample/request-generator.ts) for ES6/TypeScript).
+
+#### `agentKeepAlive`
+
+Use an agent with 'Connection: Keep-alive'.
+
+Note: Uses [agentkeepalive](https://npmjs.org/package/agentkeepalive),
+which performs better than the default node.js agent.
+
+#### `quiet`
+
+Do not show any messages.
+
+#### `indexParam`
+
+The given string will be replaced in the final URL with a unique index.
+E.g.: if URL is `http://test.com/value` and `indexParam=value`, then the URL
+will be:
+
+* http://test.com/1
+* http://test.com/2
+* ...
+* body will also be replaced `body:{ userid: id_value }` will be `body:{ userid: id_1 }`
+
+#### `indexParamCallback`
+
+A function that would be executed to replace the value identified through `indexParam` through a custom value generator.
+
+E.g.: if URL is `http://test.com/value` and `indexParam=value` and
+```javascript
+indexParamCallback: function customCallBack() {
+  return Math.floor(Math.random() * 10); //returns a random integer from 0 to 9
+}
+``` 
+then the URL could be:
+
+* http://test.com/1 (Randomly generated integer 1)
+* http://test.com/5 (Randomly generated integer 5)
+* http://test.com/6 (Randomly generated integer 6)
+* http://test.com/8 (Randomly generated integer 8)
+* ...
+* body will also be replaced `body:{ userid: id_value }` will be `body:{ userid: id_<value from callback> }`
+
+#### `insecure`
+
+Allow invalid and self-signed certificates over https.
+
+#### `secureProtocol`
+
+The TLS/SSL method to use. (e.g. TLSv1_method)
+
+Example:
+
+```javascript
+import {loadTest} from 'loadtest'
+
+const options = {
+	url: 'https://www.example.com',
+    maxRequests: 100,
+    secureProtocol: 'TLSv1_method'
+}
+
+loadTest(options, function(error) {
+	if (error) {
+		return console.error('Got an error: %s', error)
+	}
+	console.log('Tests run successfully')
+})
+```
+
+#### `statusCallback(error, result)`
+
+If present, this function executes after every request operation completes. Provides immediate access to the test result while the
+test batch is still running. This can be used for more detailed custom logging or developing your own spreadsheet or
+statistical analysis of the result.
+
+The `error` and `result` passed to the callback are in the same format as the result passed to the final callback:
+
+* `error` is only populated if the request finished in error,
+* `result` contains info about the current request: `host`, `path`, `method`, `statusCode`, received `body` and `headers`.
+Additionally has the following parameters:
+  - `requestElapsed`: time in milliseconds it took to complete this individual request.
+  - `requestIndex`: 0-based index of this particular request in the sequence of all requests to be made.
+  - `instanceIndex`: the `loadtest(...)` instance index. This is useful if you call `loadtest()` more than once.
+
+Example result:
+
+```javascript
+{
+	host: 'localhost',
+	path: '/',
+	method: 'GET',
+	statusCode: 200,
+	body: '<html><body>hi</body></html>',
+	headers: [...],
+	requestElapsed: 248,
+	requestIndex: 8748,
+	instanceIndex: 5,
+}
+```
+
+See [full example](./status-callback.md).
+
+**Warning**: The format for `statusCallback` has changed in version 7+.
+Used to be `statusCallback(error, result, latency)`;
+the third parameter `latency` has been removed due to performance reasons.
+
+#### `contentInspector(result)`
+
+A function that would be executed after every request before its status be added to the final statistics.
+
+The is can be used when you want to mark some result with 200 http status code to be failed or error.
+
+The `result` object passed to this callback function has the same fields as the `result` object passed to `statusCallback`.
+
+`customError` can be added to mark this result as failed or error. `customErrorCode` will be provided in the final statistics, in addtion to the http status code.
+
+Example:
+
+```javascript
+function contentInspector(result) {
+    if (result.statusCode == 200) {
+        const body = JSON.parse(result.body)
+        // how to examine the body depends on the content that the service returns
+        if (body.status.err_code !== 0) {
+            result.customError = body.status.err_code + " " + body.status.msg
+            result.customErrorCode = body.status.err_code
+        }
+    }
+},
+```
+    
+### Start Test Server
+
+To start the test server use the exported function `startServer()` with a set of options:
+
+```javascript
+import {startServer} from 'loadtest'
+const server = await startServer({port: 8000})
+// do your thing
+await server.close()
+```
+
+This function returns when the server is up and running,
+with an HTTP server which can be `close()`d when it is no longer useful.
+As a legacy from before promises existed,
+if an optional callback is passed as second parameter then it will not behave as `async`:
+
+```javascript
+const server = startServer({port: 8000}, error => console.error(error))
+```
+
+The following options are available.
+
+#### `port`
+
+Optional port to use for the server.
+
+Note: the default port is 7357, since port 80 requires special privileges.
+
+#### `delay`
+
+Wait the given number of milliseconds to answer each request.
+
+#### `error`
+
+Return an HTTP error code.
+
+#### `percent`
+
+Return an HTTP error code only for the given % of requests.
+If no error code was specified, default is 500.
+
+#### `logger(request, response)`
+
+A function to be called after every request served by the test server.
+`request` and `response` are the usual HTTP objects.
+

package/doc/status-callback.md

@@ -0,0 +1,105 @@
+# `statusCallback`
+
+In-depth examples for the `statusCallback` API parameter.
+
+## `options.statusCallback(error, result)`
+
+This function, if present, is invoked after every request is finished.
+It uses the old-style convention `(error, result)`:
+the `error` is only present if the request fails,
+while the `result` is always present and contains info about the request.
+
+**Warning**: The format for `statusCallback` has changed in version 7+.
+The third parameter `latency` has been removed due to performance reasons.
+
+**Warning**: The format for `statusCallback` has changed in version 2.0.0 onwards.
+It used to be `statusCallback(latency, result, error)`,
+it has been changed to conform to the usual Node.js standard.
+
+### `result` format
+
+The `result` parameter has the following attributes, always present:
+
+* `host`: the host where the request was sent.
+* `path`: the URL path to send the request.
+* `method: HTTP method used.
+* `statusCode: HTTP status code, 200 is OK.
+* `body: content received from the server.
+* `headers: sent by the server.
+* `requestElapsed`: time in milliseconds it took to complete this individual request.
+* `requestIndex`: 0-based index of this particular request in the sequence of all requests to be made.
+* `instanceIndex`: the `loadtest(...)` instance index. This is useful if you call `loadtest()` more than once.
+
+### Example Result
+
+A sample result might look like this:
+
+```javascript
+{
+	host: 'localhost',
+	path: '/',
+	method: 'GET',
+	statusCode: 200,
+	body: '<html><body>hi</body></html>',
+	headers: [...],
+	requestElapsed: 248,
+	requestIndex: 8748,
+	instanceIndex: 5,
+}
+```
+
+## Full example
+
+A full example of how to use the `statusCallback` follows:
+
+```javascript
+import {loadTest} from 'loadtest'
+
+function statusCallback(error, result, latency) {
+    console.log('Current latency %j, result %j, error %j', latency, result, error)
+    console.log('----')
+    console.log('Request elapsed milliseconds: ', result.requestElapsed)
+    console.log('Request index: ', result.requestIndex)
+    console.log('Request loadtest() instance index: ', result.instanceIndex)
+}
+
+const options = {
+    url: 'http://localhost:8000',
+    maxRequests: 1000,
+    statusCallback: statusCallback
+}
+
+loadTest(options, function(error) {
+    if (error) {
+        return console.error('Got an error: %s', error)
+    }
+    console.log('Tests run successfully')
+})
+```
+
+### Adding Request Data
+ 
+In some situations request data needs to be available in the statusCallBack.
+This data can be assigned to `request.labels` in the requestGenerator:
+```javascript
+const options = {
+	// ...
+	requestGenerator: (params, options, client, callback) => {
+		// ...
+        const randomInputData = Math.random().toString().substr(2, 8);
+        const message = JSON.stringify({ randomInputData })
+		const request = client(options, callback);
+        request.labels = randomInputData;
+		request.write(message);
+		return request;
+	}
+};
+```
+
+Then in statusCallBack the labels can be accessed through `result.labels`:
+```javascript
+function statusCallback(error, result, latency) {
+    console.log(result.labels);
+}
+```
+

package/lib/httpClient.js

@@ -9,6 +9,8 @@ import * as agentkeepalive from 'agentkeepalive'
 import * as HttpsProxyAgent from 'https-proxy-agent'
 
 
+let uniqueIndex = 1
+
 /**
  * Create a new HTTP client.
  * Seem parameters below.
@@ -90,6 +92,19 @@ class HttpClient {
 		if (this.params.secureProtocol) {
 			this.options.secureProtocol = this.params.secureProtocol;
 		}
+		// adding proxy configuration
+		if (this.params.proxy) {
+			const proxy = this.params.proxy;
+			const agent = new HttpsProxyAgent(proxy);
+			this.options.agent = agent;
+		}
+		if (this.params.indexParam) {
+			this.options.indexParamFinder = new RegExp(this.params.indexParam, 'g');
+		}
+		// Disable certificate checking
+		if (this.params.insecure === true) {
+			process.env.NODE_TLS_REJECT_UNAUTHORIZED = '0';
+		}
 	}
 
 	/**
@@ -129,43 +144,10 @@ class HttpClient {
 		this.operation.requests += 1;
 
 		const id = this.operation.latency.start();
+		const options = {...this.options, headers: {...this.options.headers}}
 		const requestFinished = this.getRequestFinisher(id);
-		let lib = http;
-		if (this.options.protocol == 'https:') {
-			lib = https;
-		}
-		if (this.options.protocol == 'ws:') {
-			lib = websocket;
-		}
-
-		// adding proxy configuration
-		if (this.params.proxy) {
-			const proxy = this.params.proxy;
-			const agent = new HttpsProxyAgent(proxy);
-			this.options.agent = agent;
-		}
-
-
-		// Disable certificate checking
-		if (this.params.insecure === true) {
-			process.env.NODE_TLS_REJECT_UNAUTHORIZED = '0';
-		}
-		let request, message;
-		if (this.generateMessage) {
-			message = this.generateMessage(id);
-			if (typeof message === 'object') {
-				message = JSON.stringify(message);
-			}
-			this.options.headers['Content-Length'] = Buffer.byteLength(message);
-		} else {
-			delete this.options.headers['Content-Length'];
-		}
-		if (typeof this.params.requestGenerator == 'function') {
-			const connect = this.getConnect(id, requestFinished, this.params.contentInspector)
-			request = this.params.requestGenerator(this.params, this.options, lib.request, connect);
-		} else {
-			request = lib.request(this.options, this.getConnect(id, requestFinished, this.params.contentInspector));
-		}
+		this.customizeIndex(options)
+		const request = this.getRequest(id, options, requestFinished)
 		if (this.params.timeout) {
 			const timeout = parseInt(this.params.timeout);
 			if (!timeout) {
@@ -175,8 +157,10 @@ class HttpClient {
 				requestFinished('Connection timed out');
 			});
 		}
+		const message = this.getMessage(id, options)
 		if (message) {
 			request.write(message);
+			options.headers['Content-Length'] = Buffer.byteLength(message);
 		}
 		request.on('error', error => {
 			requestFinished('Connection error: ' + error.message);
@@ -184,6 +168,54 @@ class HttpClient {
 		request.end();
 	}
 
+	customizeIndex(options) {
+		if (!this.options.indexParamFinder) {
+			return
+		}
+		options.customIndex = this.getCustomIndex()
+		options.path = this.options.path.replace(this.options.indexParamFinder, options.customIndex);
+	}
+
+	getCustomIndex() {
+		if (this.options.indexParamCallback instanceof Function) {
+			return this.options.indexParamCallback();
+		}
+		const customIndex = uniqueIndex
+		uniqueIndex += 1
+		return customIndex
+	}
+
+	getLib() {
+		if (this.options.protocol == 'https:') {
+			return https;
+		}
+		if (this.options.protocol == 'ws:') {
+			return websocket;
+		}
+		return http;
+	}
+
+	getMessage(id, options) {
+		if (!this.generateMessage) {
+			return
+		}
+		const candidate = this.generateMessage(id);
+		const message = typeof candidate === 'object' ? JSON.stringify(candidate) : candidate
+		if (this.options.indexParamFinder) {
+			return message.replace(this.options.indexParamFinder, options.customIndex);
+		}
+		return message
+	}
+
+	getRequest(id, options, requestFinished) {
+		const lib = this.getLib()
+		if (typeof this.params.requestGenerator == 'function') {
+			const connect = this.getConnect(id, requestFinished, this.params.contentInspector)
+			return this.params.requestGenerator(this.params, options, lib.request, connect);
+		}
+		return lib.request(options, this.getConnect(id, requestFinished, this.params.contentInspector));
+	}
+
 	/**
 	 * Get a function that finishes one request and goes for the next.
 	 */

package/lib/loadtest.js

@@ -16,23 +16,25 @@ https.globalAgent.maxSockets = 1000;
  * Run a load test.
  * Parameters:
  * - `options`: an object which may have:
- *	 - url [string]: URL to access (mandatory).
- *	 - concurrency [number]: how many concurrent clients to use.
- *	 - maxRequests [number]: how many requests to send
- *	 - maxSeconds [number]: how long to run the tests.
- *	 - cookies [array]: a string or an array of strings, each with name:value.
- *	 - headers [map]: a map with headers: {key1: value1, key2: value2}.
- *	 - method [string]: the method to use: POST, PUT. Default: GET, what else.
- *	 - body [string]: the contents to send along a POST or PUT request.
- *	 - contentType [string]: the MIME type to use for the body, default text/plain.
- *	 - requestsPerSecond [number]: how many requests per second to send.
+ *	 - url: URL to access (mandatory).
+ *	 - concurrency: how many concurrent clients to use.
+ *	 - maxRequests: how many requests to send
+ *	 - maxSeconds: how long to run the tests.
+ *	 - cookies: a string or an array of strings, each with name:value.
+ *	 - headers: a map with headers: {key1: value1, key2: value2}.
+ *	 - method: the method to use: POST, PUT. Default: GET, what else.
+ *	 - data: the contents to send along a POST or PUT request.
+ *	 - contentType: the MIME type to use for the body, default text/plain.
+ *	 - requestsPerSecond: how many requests per second to send.
  *	 - agentKeepAlive: if true, then use connection keep-alive.
- *	 - indexParam [string]: string to replace with a unique index.
+ *	 - indexParam: string to replace with a unique index.
  *	 - insecure: allow https using self-signed certs.
- *	 - secureProtocol [string]: TLS/SSL secure protocol method to use.
- *	 - proxy [string]: use a proxy for requests e.g. http://localhost:8080.
+ *	 - secureProtocol: TLS/SSL secure protocol method to use.
+ *	 - proxy: use a proxy for requests e.g. http://localhost:8080.
  *	 - quiet: do not log any messages.
  *	 - debug: show debug messages (deprecated).
+ *	 - requestGenerator: use a custom function to generate requests.
+ *	 - statusCallback: function called after every request.
  * - `callback`: optional `function(result, error)` called if/when the test finishes;
  * if not present a promise is returned.
  */
@@ -109,7 +111,7 @@ class Operation {
 			next();
 		}
 		if (this.options.statusCallback) {
-			this.options.statusCallback(error, result, this.latency.getResult());
+			this.options.statusCallback(error, result);
 		}
 	}
 
@@ -117,33 +119,9 @@ class Operation {
 	 * Start a number of measuring clients.
 	 */
 	startClients() {
-		const url = this.options.url;
-		const strBody = JSON.stringify(this.options.body);
 		for (let index = 0; index < this.options.concurrency; index++) {
-			if (this.options.indexParam) {
-				let oldToken = new RegExp(this.options.indexParam, 'g');
-				if(this.options.indexParamCallback instanceof Function) {
-					let customIndex = this.options.indexParamCallback();
-					this.options.url = url.replace(oldToken, customIndex);
-					if(this.options.body) {
-						let body = strBody.replace(oldToken, customIndex);
-						this.options.body = JSON.parse(body);
-					}
-				}
-				else {
-					this.options.url = url.replace(oldToken, index);
-					if(this.options.body) {
-						let body = strBody.replace(oldToken, index);
-						this.options.body = JSON.parse(body);
-					}
-				}
-			}
-			let constructor = httpClient.create;
-			// TODO: || this.options.url.startsWith('wss:'))
-			if (this.options.url.startsWith('ws:')) {
-				constructor = websocket.create;
-			}
-			const client = constructor(this, this.options);
+			const createClient = this.getClientCreator()
+			const client = createClient(this, this.options);
 			this.clients[index] = client;
 			if (!this.options.requestsPerSecond) {
 				client.start();
@@ -155,6 +133,14 @@ class Operation {
 		}
 	}
 
+	getClientCreator() {
+		// TODO: || this.options.url.startsWith('wss:'))
+		if (this.options.url.startsWith('ws:')) {
+			return websocket.create;
+		}
+		return httpClient.create;
+	}
+
 	/**
 	 * Stop clients.
 	 */

package/lib/options.js

@@ -45,6 +45,7 @@ class Options {
 		this.recover = options.recover || configuration.recover
 		this.proxy = options.proxy || configuration.proxy
 		this.quiet = options.quiet || configuration.quiet
+		this.statusCallback = options.statusCallback
 	}
 
 	getUrl(options) {

package/lib/testserver.js

@@ -16,6 +16,7 @@ const LOG_HEADERS_INTERVAL_MS = 5000;
  *	 - quiet: do not log any messages.
  *	 - percent: give an error (default 500) on some % of requests.
  *	 - error: set an HTTP error code, default is 500.
+ *	 - logger: function to log all incoming requests.
  * - `callback`: optional callback, called after the server has started.
  *   If not present will return a promise.
  */
@@ -118,10 +119,10 @@ class TestServer {
 				this.debug(request, elapsedMs);
 			}
 			if (!this.options.delay) {
-				return this.end(response, id);
+				return this.end(request, response, id);
 			}
 			setTimeout(() => {
-				this.end(response, id);
+				this.end(request, response, id);
 			}, this.options.delay).unref();
 		});
 	}
@@ -167,7 +168,7 @@ class TestServer {
 	/**
 	 * End the response now.
 	 */
-	end(response, id) {
+	end(request, response, id) {
 		if (this.shouldError()) {
 			const code = this.options.error || 500;
 			response.writeHead(code);
@@ -176,6 +177,9 @@ class TestServer {
 			response.end('OK');
 		}
 		this.latency.end(id);
+		if (this.options.logger) {
+			this.options.logger(request, response)
+		}
 	}
 
 	shouldError() {

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "loadtest",
-	"version": "6.3.2",
+	"version": "7.0.0",
 	"type": "module",
 	"description": "Run load tests for your web application. Mostly ab-compatible interface, with an option to force requests per second. Includes an API for automated load testing.",
 	"homepage": "https://github.com/alexfernandez/loadtest",

package/test/integration.js

@@ -159,12 +159,62 @@ async function testPromise() {
 	return 'Test result: ' + JSON.stringify(result)
 }
 
+async function testIndexParam() {
+	const urls = new Map()
+	const bodies = new Map()
+	function logger(request) {
+		if (urls.has(request.url)) {
+			throw new Error(`Duplicated url ${request.url}`)
+		}
+		urls.set(request.url, true)
+		if (bodies.has(request.body)) {
+			throw new Error(`Duplicated body ${request.body}`)
+		}
+		bodies.set(request.body, true)
+	}
+	const server = await startServer({logger, ...serverOptions})
+	const options = {
+		url: `http://localhost:${PORT}/?param=index`,
+		maxRequests: 100,
+		concurrency: 10,
+		postBody: {
+			hi: 'my_index',
+		},
+		indexParam: 'index',
+		quiet: true,
+	};
+	await loadTest(options)
+	await server.close()
+}
+
+async function testStatusCallback() {
+	let calls = 0
+	const server = await startServer(serverOptions)
+	const options = {
+		url: `http://localhost:${PORT}/`,
+		maxRequests: 100,
+		concurrency: 10,
+		postBody: {
+			hi: 'hey',
+		},
+		quiet: true,
+		statusCallback: (error, result) => {
+			testing.assertEquals(result.statusCode, 200, 'Should receive status 200')
+			calls += 1
+		}
+	};
+	await loadTest(options)
+	testing.assertEquals(calls, 100, 'Should have 100 calls')
+	await server.close()
+}
+
 /**
  * Run all tests.
  */
 export function test(callback) {
 	testing.run([
-		testIntegration, testIntegrationFile, testDelay, testWSIntegration, testPromise,
+		testIntegration, testIntegrationFile, testDelay, testWSIntegration,
+		testPromise, testIndexParam, testStatusCallback,
 	], 4000, callback);
 }