loadtest 6.0.0 → 6.2.0

package/README.md

@@ -84,7 +84,7 @@ but the resulting figure is much more robust.
 `loadtest` is also quite extensible.
 Using the provided API it is very easy to integrate loadtest with your package, and run programmatic load tests.
 loadtest makes it very easy to run load tests as part of systems tests, before deploying a new version of your software.
-The results include mean response times and percentiles,
+The result includes mean response times and percentiles,
 so that you can abort deployment e.g. if 99% of the requests don't finish in 10 ms or less.
 
 ### Usage Don'ts
@@ -111,6 +111,7 @@ The following parameters are compatible with Apache ab.
 #### `-n requests`
 
 Number of requests to send out.
+Default is no limit; will keep on sending if not specified.
 
 Note: the total number of requests sent can be bigger than the parameter if there is a concurrency parameter;
 loadtest will report just the first `n`.
@@ -119,6 +120,7 @@ loadtest will report just the first `n`.
 
 loadtest will create a certain number of clients; this parameter controls how many.
 Requests from them will arrive concurrently to the server.
+Default value is 1.
 
 Note: requests are not sent in parallel (from different processes),
 but concurrently (a second request may be sent before the first has been answered).
@@ -126,6 +128,7 @@ but concurrently (a second request may be sent before the first has been answere
 #### `-t timelimit`
 
 Max number of seconds to wait until requests no longer go out.
+Default is no limit; will keep on sending if not specified.
 
 Note: this is different than Apache `ab`, which stops _receiving_ requests after the given seconds.
 
@@ -175,15 +178,17 @@ Send the string as the PATCH body. E.g.: `-A '{"key": "a9acf03f"}'`
 
 #### `-m method`
 
-Send method to link. Accept: [GET, POST, PUT, DELETE, PATCH, get, post, put, delete, patch], Default is GET
-E.g.: -m POST
+Set method that will be sent to the test URL.
+Accepts: `GET`, `POST`, `PUT`, `DELETE`, `PATCH`,
+and lowercase versions. Default is `GET`.
+Example: `-m POST`.
 
-#### `--data POST some variables`
+#### `--data body`
 
-Send some data. It does not support method GET.
-E.g: `--data '{"username": "test", "password": "test"}' -T 'application/x-www-form-urlencoded' -m POST`
+Add some data to send in the body. It does not support method GET.
+Requires setting the method with `-m` and the type with `-T`.
+Example: `--data '{"username": "test", "password": "test"}' -T 'application/x-www-form-urlencoded' -m POST`
 
-It required `-m` and `-T 'application/x-www-form-urlencoded'`
 
 #### `-p POST-file`
 
@@ -256,6 +261,7 @@ The following parameters are _not_ compatible with Apache ab.
 
 Controls the number of requests per second that are sent.
 Can be fractional, e.g. `--rps 0.5` sends one request every two seconds.
+Not used by default: each request is sent as soon as the previous one is responded.
 
 Note: Concurrency doesn't affect the final number of requests per second,
 since rps will be shared by all the clients. E.g.:
@@ -309,17 +315,15 @@ Open connections using keep-alive.
 
 Note: instead of using the default agent, this option is now an alias for `-k`.
 
-#### `--quiet` (deprecated)
+#### `--quiet`
 
 Do not show any messages.
 
-Note: deprecated in version 6+, shows a warning.
-
 #### `--debug` (deprecated)
 
 Show debug messages.
 
-Note: deprecated in version 6+, shows a warning.
+Note: deprecated in version 6+.
 
 #### `--insecure`
 
@@ -383,7 +387,7 @@ with concurrency 10 (only relevant results are shown):
       99%      14 ms
      100%      35997 ms (longest request)
 
-Results were quite erratic, with some requests taking up to 36 seconds;
+The result was quite erratic, with some requests taking up to 36 seconds;
 this suggests that Node.js is queueing some requests for a long time, and answering them irregularly.
 Now we will try a fixed rate of 1000 rps:
 
@@ -438,10 +442,10 @@ We now know that our server can accept 500 rps without problems.
 Not bad for a single-process naïve Node.js server...
 We may refine our results further to find at which point from 500 to 1000 rps our server breaks down.
 
-But instead let us research how to improve the results.
+But instead let us research how to improve the result.
 One obvious candidate is to add keep-alive to the requests so we don't have to create
 a new connection for every request.
-The results (with the same test server) are impressive:
+The result (with the same test server) is impressive:
 
     $ loadtest http://localhost:7357/ -t 20 -c 10 -k
     ...
@@ -477,7 +481,28 @@ thus allowing you to load test your application in your own tests.
 
 ### Invoke Load Test
 
-To run a load test, just call the exported function `loadTest()` with a set of options and an optional callback:
+To run a load test, just `await` for the exported function `loadTest()` with the desired options, described below:
+
+```javascript
+import {loadTest} from 'loadtest'
+
+const options = {
+	url: 'http://localhost:8000',
+	maxRequests: 1000,
+}
+const result = await loadTest(options)
+result.show()
+console.log('Tests run successfully')
+})
+```
+
+The call returns a `Result` object that contains all info about the load test, also described below.
+Call `result.show()` to display the results in the standard format on the console.
+
+As a legacy from before promises existed,
+if an optional callback is passed as second parameter then it will not behave as `async`:
+the callback `function(error, result)` will be invoked when the max number of requests is reached,
+or when the max number of seconds has elapsed.
 
 ```javascript
 import {loadTest} from 'loadtest'
@@ -490,16 +515,51 @@ loadTest(options, function(error, result) {
 	if (error) {
 		return console.error('Got an error: %s', error)
 	}
+	result.show()
 	console.log('Tests run successfully')
 })
 ```
 
-The callback `function(error, result)` will be invoked when the max number of requests is reached,
-or when the max number of seconds has elapsed.
 
 Beware: if there are no `maxRequests` and no `maxSeconds`, then tests will run forever
 and will not call the callback.
 
+### Result
+
+The latency result returned at the end of the load test contains a full set of data, including:
+mean latency, number of errors and percentiles.
+An example follows:
+
+```javascript
+{
+  url: 'http://localhost:80/',
+  maxRequests: 1000,
+  maxSeconds: 0,
+  concurrency: 10,
+  agent: 'none',
+  requestsPerSecond: undefined,
+  totalRequests: 1000,
+  percentiles: {
+	'50': 7,
+	'90': 10,
+	'95': 11,
+	'99': 15
+  },
+  rps: 2824,
+  totalTimeSeconds: 0.354108,
+  meanLatencyMs: 7.72,
+  maxLatencyMs: 20,
+  totalErrors: 3,
+  errorCodes: {
+	'0': 1,
+	'500': 2
+  },
+}
+```
+
+The `result` object also has a `result.show()` function
+that displays the results on the console in the standard format.
+
 ### Options
 
 All options but `url` are, as their name implies, optional.
@@ -587,6 +647,9 @@ function(params, options, client, callback) {
 }
 ```
 
+See [`sample/request-generator.js`](sample/request-generator.js) for some sample code including a body
+(or [`sample/request-generator.ts`](sample/request-generator.ts) for ES6/TypeScript).
+
 #### `agentKeepAlive`
 
 Use an agent with 'Connection: Keep-alive'.
@@ -659,11 +722,11 @@ loadTest(options, function(error) {
 
 #### `statusCallback`
 
-If present, this function executes after every request operation completes. Provides immediate access to test results while the
+If present, this function executes after every request operation completes. Provides immediate access to the test result while the
 test batch is still running. This can be used for more detailed custom logging or developing your own spreadsheet or
-statistical analysis of results.
+statistical analysis of the result.
 
-The results and error passed to the callback are in the same format as the results passed to the final callback.
+The result and error passed to the callback are in the same format as the result passed to the final callback.
  
 In addition, the following three properties are added to the `result` object:
 
@@ -673,6 +736,19 @@ In addition, the following three properties are added to the `result` object:
 
 You will need to check if `error` is populated in order to determine which object to check for these properties.
 
+The second parameter contains info about the current request:
+
+```javascript
+{
+	host: 'localhost',
+	path: '/',
+	method: 'GET',
+	statusCode: 200,
+	body: '<html><body>hi</body></html>',
+	headers: [...]
+}
+```
+
 Example:
 
 ```javascript
@@ -699,7 +775,6 @@ loadTest(options, function(error) {
     console.log('Tests run successfully')
 })
 ```
-
  
 In some situations request data needs to be available in the statusCallBack.
 This data can be assigned to `request.labels` in the requestGenerator:
@@ -753,57 +828,26 @@ function contentInspector(result) {
     }
 },
 ```
-
-### Results
-
-The latency results passed to your callback at the end of the load test contains a full set of data, including:
-mean latency, number of errors and percentiles.
-An example follows:
-
-```javascript
-{
-  totalRequests: 1000,
-  percentiles: {
-	'50': 7,
-	'90': 10,
-	'95': 11,
-	'99': 15
-  },
-  rps: 2824,
-  totalTimeSeconds: 0.354108,
-  meanLatencyMs: 7.72,
-  maxLatencyMs: 20,
-  totalErrors: 3,
-  errorCodes: {
-	'0': 1,
-	'500': 2
-  }
-}
-```
-
-The second parameter contains info about the current request:
-
-```javascript
-{
-	host: 'localhost',
-	path: '/',
-	method: 'GET',
-	statusCode: 200,
-	body: '<html><body>hi</body></html>',
-	headers: [...]
-}
-```
     
 ### Start Test Server
 
-To start the test server use the exported function `startServer()` with a set of options and an optional callback:
+To start the test server use the exported function `startServer()` with a set of options:
 
 ```javascript
 import {startServer} from 'loadtest'
-const server = startServer({port: 8000})
+const server = await startServer({port: 8000})
+// do your thing
+await server.close()
 ```
 
-This function returns an HTTP server which can be `close()`d when it is no longer useful.
+This function returns when the server is up and running,
+with an HTTP server which can be `close()`d when it is no longer useful.
+As a legacy from before promises existed,
+if an optional callback is passed as second parameter then it will not behave as `async`:
+
+```
+const server = startServer({port: 8000}, error => console.error(error))
+```
 
 The following options are available.
 

package/bin/loadtest.js

@@ -31,8 +31,8 @@ const options = stdio.getopt({
 	insecure: {description: 'Allow self-signed certificates over https'},
 	key: {args: 1, description: 'The client key to use'},
 	cert: {args: 1, description: 'The client certificate to use'},
+	quiet: {description: 'Do not log any messages'},
 	agent: {description: 'Use a keep-alive http agent (deprecated)'},
-	quiet: {description: 'Do not log any messages (deprecated)'},
 	debug: {description: 'Show debug messages (deprecated)'}
 });
 
@@ -52,7 +52,8 @@ async function processAndRun(options) {
 	}
 	options.url = options.args[0];
 	try {
-		loadTest(options)
+		const result = await loadTest(options)
+		result.show()
 	} catch(error) {
 		console.error(error.message)
 		help()

package/lib/httpClient.js

@@ -28,6 +28,7 @@ class HttpClient {
 	constructor(operation, params) {
 		this.operation = operation
 		this.params = params
+		this.stopped = false
 		this.init();
 	}
 
@@ -45,11 +46,15 @@ class HttpClient {
 			this.options.key = this.params.key;
 		}
 		this.options.agent = false;
+		if (this.params.requestsPerSecond) {
+			// rps for each client is total / concurrency (# of clients)
+			this.options.requestsPerSecond = this.params.requestsPerSecond / this.params.concurrency
+		}
 		if (this.params.agentKeepAlive) {
-			const KeepAlive = (this.options.protocol == 'https:') ? agentkeepalive.HttpsAgent : agentkeepalive;
+			const KeepAlive = (this.options.protocol == 'https:') ? agentkeepalive.HttpsAgent : agentkeepalive.default;
 			let maxSockets = 10;
-			if (this.params.requestsPerSecond) {
-				maxSockets += Math.floor(this.params.requestsPerSecond);
+			if (this.options.requestsPerSecond) {
+				maxSockets += Math.floor(this.options.requestsPerSecond);
 			}
 			this.options.agent = new KeepAlive({
 				maxSockets: maxSockets,
@@ -71,7 +76,7 @@ class HttpClient {
 			} else if (typeof this.params.body == 'function') {
 				this.generateMessage = this.params.body;
 			} else {
-				console.error('Unrecognized body: %s', typeof this.params.body);
+				throw new Error(`Unrecognized body: ${typeof this.params.body}`);
 			}
 			this.options.headers['Content-Type'] = this.params.contentType || 'text/plain';
 		}
@@ -81,7 +86,7 @@ class HttpClient {
 			} else if (typeof this.params.cookies == 'string') {
 				this.options.headers.Cookie = this.params.cookies;
 			} else {
-				console.error('Invalid cookies %j, please use an array or a string', this.params.cookies);
+				throw new Error(`Invalid cookies ${JSON.stringify(this.params.cookies)}, please use an array or a string`);
 			}
 		}
 		addUserAgent(this.options.headers);
@@ -94,10 +99,15 @@ class HttpClient {
 	 * Start the HTTP client.
 	 */
 	start() {
-		if (!this.params.requestsPerSecond) {
+		if (this.stopped) {
+			// solves testing issue: with requestsPerSecond clients are started at random,
+			// so sometimes they are stopped before they have even started
+			return
+		}
+		if (!this.options.requestsPerSecond) {
 			return this.makeRequest();
 		}
-		const interval = 1000 / this.params.requestsPerSecond;
+		const interval = 1000 / this.options.requestsPerSecond;
 		this.requestTimer = new HighResolutionTimer(interval, () => this.makeRequest());
 	}
 
@@ -105,6 +115,7 @@ class HttpClient {
 	 * Stop the HTTP client.
 	 */
 	stop() {
+		this.stopped = true
 		if (this.requestTimer) {
 			this.requestTimer.stop();
 		}
@@ -133,7 +144,6 @@ class HttpClient {
 		// adding proxy configuration
 		if (this.params.proxy) {
 			const proxy = this.params.proxy;
-			//console.log('using proxy server %j', proxy);
 			const agent = new HttpsProxyAgent(proxy);
 			this.options.agent = agent;
 		}
@@ -154,7 +164,8 @@ class HttpClient {
 			delete this.options.headers['Content-Length'];
 		}
 		if (typeof this.params.requestGenerator == 'function') {
-			request = this.params.requestGenerator(this.params, this.options, lib.request, this.getConnect(id, requestFinished, this.params.contentInspector));
+			const connect = this.getConnect(id, requestFinished, this.params.contentInspector)
+			request = this.params.requestGenerator(this.params, this.options, lib.request, connect);
 		} else {
 			request = lib.request(this.options, this.getConnect(id, requestFinished, this.params.contentInspector));
 		}
@@ -205,7 +216,7 @@ class HttpClient {
 				result.instanceIndex = this.operation.instanceIndex;
 			}
 			let callback;
-			if (!this.params.requestsPerSecond) {
+			if (!this.options.requestsPerSecond) {
 				callback = this.makeRequest.bind(this);
 			}
 			this.operation.callback(error, result, callback);

package/lib/latency.js

@@ -1,12 +1,13 @@
 import * as crypto from 'crypto'
+import {Result} from './result.js'
 
 
 /**
  * Latency measurements. Options can be:
  *	- maxRequests: max number of requests to measure before stopping.
  *	- maxSeconds: max seconds, alternative to max requests.
- * An optional callback(error, results) will be called with an error,
- * or the results after max is reached.
+ * An optional callback(error, result) will be called with an error,
+ * or the result after max is reached.
  */
 export class Latency {
 	constructor(options, callback) {
@@ -106,7 +107,7 @@ export class Latency {
 	showPartial() {
 		const elapsedSeconds = this.getElapsed(this.lastShown) / 1000;
 		const meanTime = this.partialTime / this.partialRequests || 0.0;
-		const results = {
+		const result = {
 			meanLatencyMs: Math.round(meanTime * 10) / 10,
 			rps: Math.round(this.partialRequests / elapsedSeconds)
 		};
@@ -114,10 +115,12 @@ export class Latency {
 		if (this.options.maxRequests) {
 			percent = ' (' + Math.round(100 * this.totalRequests / this.options.maxRequests) + '%)';
 		}
-		console.info('Requests: %s%s, requests per second: %s, mean latency: %s ms', this.totalRequests, percent, results.rps, results.meanLatencyMs);
-		if (this.totalErrors) {
-			percent = Math.round(100 * 10 * this.totalErrors / this.totalRequests) / 10;
-			console.info('Errors: %s, accumulated errors: %s, %s% of total requests', this.partialErrors, this.totalErrors, percent);
+		if (!this.options.quiet) {
+			console.info('Requests: %s%s, requests per second: %s, mean latency: %s ms', this.totalRequests, percent, result.rps, result.meanLatencyMs);
+			if (this.totalErrors) {
+				percent = Math.round(100 * 10 * this.totalErrors / this.totalRequests) / 10;
+				console.info('Errors: %s, accumulated errors: %s, %s% of total requests', this.partialErrors, this.totalErrors, percent);
+			}
 		}
 		this.partialTime = 0;
 		this.partialRequests = 0;
@@ -163,27 +166,16 @@ export class Latency {
 	finish() {
 		this.running = false;
 		if (this.callback) {
-			return this.callback(null, this.getResults());
+			return this.callback(null, this.getResult());
 		}
 	}
 
 	/**
-	 * Get final results.
+	 * Get final result.
 	 */
-	getResults() {
-		const elapsedSeconds = this.getElapsed(this.initialTime) / 1000;
-		const meanTime = this.totalTime / this.totalRequests;
-		return {
-			totalRequests: this.totalRequests,
-			totalErrors: this.totalErrors,
-			totalTimeSeconds: elapsedSeconds,
-			rps: Math.round(this.totalRequests / elapsedSeconds),
-			meanLatencyMs: Math.round(meanTime * 10) / 10,
-			maxLatencyMs: this.maxLatencyMs,
-			minLatencyMs: this.minLatencyMs,
-			percentiles: this.computePercentiles(),
-			errorCodes: this.errorCodes
-		};
+	getResult() {
+		const result = new Result(this.options, this)
+		return result
 	}
 
 	/**
@@ -215,57 +207,18 @@ export class Latency {
 	}
 
 	/**
-	 * Show final results.
+	 * Show final result.
 	 */
 	show() {
 		if (this.totalsShown) {
 			return;
 		}
 		this.totalsShown = true;
-		const results = this.getResults();
-		console.info('');
-		console.info('Target URL:          %s', this.options.url);
-		if (this.options.maxRequests) {
-			console.info('Max requests:        %s', this.options.maxRequests);
-		} else if (this.options.maxSeconds) {
-			console.info('Max time (s):        %s', this.options.maxSeconds);
-		}
-		console.info('Concurrency level:   %s', this.options.concurrency);
-		let agent = 'none';
-		if (this.options.agentKeepAlive) {
-			agent = 'keepalive';
-		}
-		console.info('Agent:               %s', agent);
-		if (this.options.requestsPerSecond) {
-			console.info('Requests per second: %s', this.options.requestsPerSecond * this.options.concurrency);
-		}
-		console.info('');
-		console.info('Completed requests:  %s', results.totalRequests);
-		console.info('Total errors:        %s', results.totalErrors);
-		console.info('Total time:          %s s', results.totalTimeSeconds);
-		console.info('Requests per second: %s', results.rps);
-		console.info('Mean latency:        %s ms', results.meanLatencyMs);
-		console.info('');
-		console.info('Percentage of the requests served within a certain time');
-
-		Object.keys(results.percentiles).forEach(percentile => {
-			console.info('  %s%      %s ms', percentile, results.percentiles[percentile]);
-		});
-
-		console.info(' 100%      %s ms (longest request)', this.maxLatencyMs);
-		if (results.totalErrors) {
-			console.info('');
-			console.info(' 100%      %s ms (longest request)', this.maxLatencyMs);
-			console.info('');
-			Object.keys(results.errorCodes).forEach(errorCode => {
-				const padding = ' '.repeat(errorCode.length < 4 ? 4 - errorCode.length : 1);
-				console.info(' %s%s:   %s errors', padding, errorCode, results.errorCodes[errorCode]);
-			});
-		}
+		const result = this.getResult();
+		result.show()
 	}
 }
 
-
 /**
  * Create a unique, random token.
  */

package/lib/loadtest.js

@@ -14,33 +14,49 @@ https.globalAgent.maxSockets = 1000;
 
 /**
  * Run a load test.
- * Options is an object which may have:
- *	- url: mandatory URL to access.
- *	- concurrency: how many concurrent clients to use.
- *	- maxRequests: how many requests to send
- *	- maxSeconds: how long to run the tests.
- *	- cookies: a string or an array of strings, each with name:value.
- *	- headers: a map with headers: {key1: value1, key2: value2}.
- *	- method: the method to use: POST, PUT. Default: GET, what else.
- *	- body: the contents to send along a POST or PUT request.
- *	- contentType: the MIME type to use for the body, default text/plain.
- *	- requestsPerSecond: how many requests per second to send.
- *	- agentKeepAlive: if true, then use connection keep-alive.
- *	- indexParam: string to replace with a unique index.
- *	- insecure: allow https using self-signed certs.
- *	- debug: show debug messages (deprecated).
- *	- quiet: do not log any messages (deprecated).
- * An optional callback will be called if/when the test finishes.
+ * Parameters:
+ * - `options`: an object which may have:
+ *	 - url [string]: URL to access (mandatory).
+ *	 - concurrency [number]: how many concurrent clients to use.
+ *	 - maxRequests [number]: how many requests to send
+ *	 - maxSeconds [number]: how long to run the tests.
+ *	 - cookies [array]: a string or an array of strings, each with name:value.
+ *	 - headers [map]: a map with headers: {key1: value1, key2: value2}.
+ *	 - method [string]: the method to use: POST, PUT. Default: GET, what else.
+ *	 - body [string]: the contents to send along a POST or PUT request.
+ *	 - contentType [string]: the MIME type to use for the body, default text/plain.
+ *	 - requestsPerSecond [number]: how many requests per second to send.
+ *	 - agentKeepAlive: if true, then use connection keep-alive.
+ *	 - indexParam [string]: string to replace with a unique index.
+ *	 - insecure: allow https using self-signed certs.
+ *	 - secureProtocol [string]: TLS/SSL secure protocol method to use.
+ *	 - proxy [string]: use a proxy for requests e.g. http://localhost:8080.
+ *	 - quiet: do not log any messages.
+ *	 - debug: show debug messages (deprecated).
+ * - `callback`: optional `function(result, error)` called if/when the test finishes;
+ * if not present a promise is returned.
  */
 export function loadTest(options, callback) {
-	processOptions(options, error => {
-		if (error) {
-			if (callback) return callback(error)
-			throw new error
-		}
-		const operation = new Operation(options, callback);
+	if (!callback) {
+		return loadTestAsync(options)
+	}
+	loadTestAsync(options).then(result => callback(null, result)).catch(error => callback(error))
+}
+
+async function loadTestAsync(options) {
+	const processed = await processOptions(options)
+	return await runOperation(processed)
+}
+
+function runOperation(options) {
+	return new Promise((resolve, reject) => {
+		const operation = new Operation(options, (error, result) => {
+			if (error) {
+				return reject(error)
+			}
+			return resolve(result)
+		});
 		operation.start();
-		return operation;
 	})
 }
 
@@ -93,7 +109,7 @@ class Operation {
 			next();
 		}
 		if (this.options.statusCallback) {
-			this.options.statusCallback(error, result, this.latency.getResults());
+			this.options.statusCallback(error, result, this.latency.getResult());
 		}
 	}
 
@@ -143,20 +159,19 @@ class Operation {
 	 * Stop clients.
 	 */
 	stop() {
+		this.running = false;
+		this.latency.running = false;
 		if (this.showTimer) {
 			this.showTimer.stop();
 		}
 		if (this.stopTimeout) {
 			clearTimeout(this.stopTimeout);
 		}
-		this.running = false;
-		this.latency.running = false;
-
 		Object.keys(this.clients).forEach(index => {
 			this.clients[index].stop();
 		});
 		if (this.finalCallback) {
-			const result = this.latency.getResults();
+			const result = this.latency.getResult();
 			result.instanceIndex = this.instanceIndex;
 			this.finalCallback(null, result);
 		} else {