loadtest 7.1.0 → 8.0.0

package/.github/FUNDING.yml

@@ -0,0 +1,13 @@
+# These are supported funding model platforms
+
+github: alexfernandez
+patreon: # Replace with a single Patreon username
+open_collective: # Replace with a single Open Collective username
+ko_fi: # Replace with a single Ko-fi username
+tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
+community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
+liberapay: # Replace with a single Liberapay username
+issuehunt: # Replace with a single IssueHunt username
+otechie: # Replace with a single Otechie username
+lfx_crowdfunding: # Replace with a single LFX Crowdfunding project-name e.g., cloud-foundry
+custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2']

package/README.md

@@ -77,7 +77,7 @@ but the resulting figure is much more robust.
 Using the provided API it is very easy to integrate loadtest with your package, and run programmatic load tests.
 loadtest makes it very easy to run load tests as part of systems tests, before deploying a new version of your software.
 The result includes mean response times and percentiles,
-so that you can abort deployment e.g. if 99% of the requests don't finish in 10 ms or less.
+so that you can abort deployment e.g. if 99% of all requests don't finish in 10 ms or less.
 
 ### Usage Don'ts
 
@@ -85,14 +85,18 @@ so that you can abort deployment e.g. if 99% of the requests don't finish in 10
 but it is still limited.
 `loadtest` saturates a single CPU pretty quickly,
 so it uses half the available cores in your processor.
-If you see that the Node.js processes are above 100% usage in `top`,
-which happens approx. when your load is above 4000~5000 rps per core,
-please adjust the number of cores.
+The Node.js processes can reach 100% usage in `top`,
+which happens approx. when your load is above 4000~7000 rps per core.
+In this case please adjust the number of cores.
 So for instance with eight cores you can expect to get a maximum performance of
-8 * 5000 ~ 40 krps.
-(You can measure the practical limits of `loadtest` on your specific test machines by running it against a simple
+8 * 5000 = 40 krps.
+
+You can measure the practical limits of `loadtest` on your specific test machines by running it against a simple
 [test server](#test-server)
-and seeing when it reaches 100% CPU.)
+and seeing when it reaches 100% CPU. Run the following commands on two different consoles:
+
+    $ node bin/testserver.js
+    $ node bin/loadtest.js -n 1000000 -c 100 http://localhost:7357/
 
 If you have reached the limits of `loadtest` even after using all cores,
 there are other tools that you can try.
@@ -100,7 +104,7 @@ there are other tools that you can try.
 * [AutoCannon](https://www.npmjs.com/package/autocannon): also an `npm` package,
 awesome tool with an interface similar to `wrk`.
 * [Apache `ab`](http://httpd.apache.org/docs/2.2/programs/ab.html)
-has great performance, but it is also limited by a single CPU performance.
+has great performance, but it is limited by a single CPU performance.
 Its practical limit is somewhere around ~40 krps.
 * [weighttp](http://redmine.lighttpd.net/projects/weighttp/wiki) is also `ab`-compatible
 and is supposed to be very fast (the author has not personally used it).
@@ -112,43 +116,57 @@ It may need installing from source though, and its interface is not `ab`-compati
 
 The following parameters are compatible with Apache ab.
 
-#### `-n requests`
+#### `-t`, `--maxSeconds`
+
+Max number of seconds to wait until requests no longer go out.
+Default is 10 seconds, applies only if no `--maxRequests` is specified.
+
+Note: this is different than Apache `ab`, which stops _receiving_ requests after the given seconds.
+
+**Warning**: max seconds used to have no default value,
+so tests would run indefinitely if no `--maxSeconds` and no `--maxRequests` were specified.
+Max seconds was changed to default to 10 in version 8.
+
+#### `-n`, `--maxRequests`
 
 Number of requests to send out.
-Default is no limit; will keep on sending if not specified.
+Default is no limit;
+will keep on sending until the time limit in `--maxSeconds` is reached.
 
 Note: the total number of requests sent can be bigger than the parameter if there is a concurrency parameter;
 loadtest will report just the first `n`.
 
-#### `-c concurrency`
+#### `-c`, `--concurrency`
 
 loadtest will create a certain number of clients; this parameter controls how many.
 Requests from them will arrive concurrently to the server.
-Default value is 1.
+Default value is 10.
 
 Note: requests are not sent in parallel (from different processes),
 but concurrently (a second request may be sent before the first has been answered).
+Does not apply if `--requestsPerSecond` is specified.
 
-#### `-t timelimit`
-
-Max number of seconds to wait until requests no longer go out.
-Default is no limit; will keep on sending if not specified.
+Beware: if concurrency is too low then it is possible that there will not be enough clients
+to send all the supported traffic,
+adjust it with `-c` if needed.
 
-Note: this is different than Apache `ab`, which stops _receiving_ requests after the given seconds.
+**Warning**: concurrency used to have a default value of 1,
+until it was changed to 10 in version 8.
 
-#### `-k` or `--keepalive`
+#### `-k`, `--keepalive`
 
-Open connections using keep-alive: use header 'Connection: Keep-alive' instead of 'Connection: Close'.
+Open connections using keep-alive:
+use header `Connection: keep-alive` instead of `Connection: close`.
 
 Note: Uses [agentkeepalive](https://npmjs.org/package/agentkeepalive),
 which performs better than the default node.js agent.
 
-#### `-C cookie-name=value`
+#### `-C`, `--cookie cookie-name=value`
 
 Send a cookie with the request. The cookie `name=value` is then sent to the server.
 This parameter can be repeated as many times as needed.
 
-#### `-H header:value`
+#### `-H`, `--header header:value`
 
 Send a custom header with the request. The line `header:value` is then sent to the server.
 This parameter can be repeated as many times as needed.
@@ -168,19 +186,19 @@ Note: if you need to add a header with spaces, be sure to surround both header a
 
     $ loadtest -H "Authorization: Basic xxx=="
 
-#### `-T content-type`
+#### `-T`, `--contentType`
 
 Set the MIME content type for POST data. Default: `text/plain`.
 
-#### `-P POST-body`
+#### `-P`, `--postBody`
 
 Send the string as the POST body. E.g.: `-P '{"key": "a9acf03f"}'`
 
-#### `-A PATCH-body`
+#### `-A`, `--patchBody`
 
 Send the string as the PATCH body. E.g.: `-A '{"key": "a9acf03f"}'`
 
-#### `-m method`
+#### `-m`, `--method`
 
 Set method that will be sent to the test URL.
 Accepts: `GET`, `POST`, `PUT`, `DELETE`, `PATCH`,
@@ -194,7 +212,7 @@ Requires setting the method with `-m` and the type with `-T`.
 Example: `--data '{"username": "test", "password": "test"}' -T 'application/x-www-form-urlencoded' -m POST`
 
 
-#### `-p POST-file`
+#### `-p`, `--postFile`
 
 Send the data contained in the given file in the POST body.
 Remember to set `-T` to the correct content-type.
@@ -218,7 +236,7 @@ export default function request(requestId) {
 
 See sample file in `sample/post-file.js`, and test in `test/body-generator.js`.
 
-#### `-u PUT-file`
+#### `-u`, `--putFile`
 
 Send the data contained in the given file as a PUT request.
 Remember to set `-T` to the correct content-type.
@@ -229,7 +247,7 @@ to provide the body of each request.
 This is useful if you want to generate request bodies dynamically and vary them for each request.
 For examples see above for `-p`.
 
-#### `-a PATCH-file`
+#### `-a`, `--patchFile`
 
 Send the data contained in the given file as a PATCH request.
 Remember to set `-T` to the correct content-type.
@@ -240,12 +258,12 @@ to provide the body of each request.
 This is useful if you want to generate request bodies dynamically and vary them for each request.
 For examples see above for `-p`.
 
-##### `-r recover`
+##### `-r`, `--recover`
 
 Recover from errors. Always active: loadtest does not stop on errors.
 After the tests are finished, if there were errors a report with all error codes will be shown.
 
-#### `-s secureProtocol`
+#### `-s`, `--secureProtocol`
 
 The TLS/SSL method to use. (e.g. TLSv1_method)
 
@@ -253,7 +271,7 @@ Example:
 
     $ loadtest -n 1000 -s TLSv1_method https://www.example.com
 
-#### `-V version`
+#### `-V`, `--version`
 
 Show version number and exit.
 
@@ -261,25 +279,17 @@ Show version number and exit.
 
 The following parameters are _not_ compatible with Apache ab.
 
-#### `--rps requestsPerSecond`
+#### `--rps`, `--requestsPerSecond`
 
 Controls the number of requests per second that are sent.
 Cannot be fractional, e.g. `--rps 0.5`.
 In this mode each request is not sent as soon as the previous one is responded,
 but periodically even if previous requests have not been responded yet.
 
-Note: Concurrency doesn't affect the final number of requests per second,
-since rps will be shared by all the clients. E.g.:
-
-    loadtest <url> -c 10 --rps 10
-
-will send a total of 10 rps to the given URL, from 10 different clients
-(each client will send 1 request per second).
-
-Beware: if concurrency is too low then it is possible that there will not be enough clients
-to send all of the rps, adjust it with `-c` if needed.
+Note: the `--concurrency` option will be ignored if `--requestsPerSecond` is specified;
+clients will be created on demand.
 
-Note: --rps is not supported for websockets.
+Note: `--rps` is not supported for websockets.
 
 #### `--cores number`
 
@@ -306,7 +316,7 @@ Setting this to 0 disables timeout (default).
 #### `-R requestGeneratorModule.js`
 
 Use a custom request generator function from an external file.
-See an example of a request generator module in [`--requestGenerator`](#requestGenerator) below.
+See an example of a request generator module in [`requestGenerator`](doc/api.md#requestGenerator).
 Also see [`sample/request-generator.js`](sample/request-generator.js) for some sample code including a body
 (or [`sample/request-generator.ts`](sample/request-generator.ts) for ES6/TypeScript).
 
@@ -338,6 +348,16 @@ Sets the certificate for the http client to use. Must be used with `--key`.
 
 Sets the key for the http client to use. Must be used with `--cert`.
 
+#### `--tcp` (experimental)
+
+Option to use low level TCP sockets,
+faster than the standard HTTP library.
+Not all options are supported.
+
+**Warning**: experimental option.
+May not work with your test case.
+See [TCP Sockets Performance](doc/tcp-sockets.md) for details.
+
 ### Test Server
 
 loadtest bundles a test server. To run it:
@@ -403,7 +423,7 @@ with concurrency 10 (only relevant results are shown):
     Requests per second: 368
     Total time:          44.503181166000005 s
 
-    Percentage of the requests served within a certain time
+    Percentage of requests served within a certain time
       50%      4 ms
       90%      5 ms
       95%      6 ms
@@ -420,7 +440,7 @@ Now we will try a fixed rate of 1000 rps:
     Requests: 9546, requests per second: 1000, mean latency: 0 ms
     Requests: 14549, requests per second: 1000, mean latency: 20 ms
     ...
-    Percentage of the requests served within a certain time
+    Percentage of requests served within a certain time
       50%      1 ms
       90%      2 ms
       95%      8 ms
@@ -451,7 +471,7 @@ Let us lower the rate to 500 rps:
     Requests per second: 488
     Total time:          20.002735398000002 s
 
-    Percentage of the requests served within a certain time
+    Percentage of requests served within a certain time
       50%      1 ms
       90%      1 ms
       95%      1 ms
@@ -474,7 +494,7 @@ The result (with the same test server) is impressive:
     ...
     Requests per second: 4099
 
-    Percentage of the requests served within a certain time
+    Percentage of requests served within a certain time
     50%      2 ms
     90%      3 ms
     95%      3 ms
@@ -487,7 +507,7 @@ Now we're talking! The steady rate also goes up to 2 krps:
     ...
     Requests per second: 1950
 
-    Percentage of the requests served within a certain time
+    Percentage of requests served within a certain time
       50%      1 ms
       90%      2 ms
       95%      2 ms
@@ -567,6 +587,8 @@ see [doc/api.md](doc/api.md) for details.
 * `percent`: return error only for the given % of requests.
 * `logger(request, response)`: function to call after every request.
 
+Returns a test server that you can `close()` when finished.
+
 ### Configuration file
 
 It is possible to put configuration options in a file named `.loadtestrc` in your working directory or in a file whose name is specified in the `loadtest` entry of your `package.json`. The options in the file will be used only if they are not specified in the command line.

package/bin/loadtest.js

@@ -9,9 +9,9 @@ import {getHalfCores} from '../lib/cluster.js'
 
 
 const options = stdio.getopt({
+	maxSeconds: {key: 't', args: 1, description: 'Max time in seconds to wait for responses, default 10'},
 	maxRequests: {key: 'n', args: 1, description: 'Number of requests to perform'},
-	concurrency: {key: 'c', args: 1, description: 'Number of requests to make'},
-	maxSeconds: {key: 't', args: 1, description: 'Max time in seconds to wait for responses'},
+	concurrency: {key: 'c', args: 1, description: 'Number of concurrent requests, default 10'},
 	timeout: {key: 'd', args: 1, description: 'Timeout for each request in milliseconds'},
 	contentType: {key: 'T', args: 1, description: 'MIME type for the body'},
 	cookies: {key: 'C', multiple: true, description: 'Send a cookie as name=value'},
@@ -36,6 +36,7 @@ const options = stdio.getopt({
 	cert: {args: 1, description: 'The client certificate to use'},
 	quiet: {description: 'Do not log any messages'},
 	cores: {args: 1, description: 'Number of cores to use', default: getHalfCores()},
+	tcp: {description: 'Use TCP sockets (experimental)'},
 	agent: {description: 'Use a keep-alive http agent (deprecated)'},
 	debug: {description: 'Show debug messages (deprecated)'},
 });

package/bin/tcp-performance.js

@@ -0,0 +1,21 @@
+import {loadTest, startServer} from '../index.js'
+
+const port = 7359;
+const serverOptions = {port}
+
+
+async function runTcpPerformanceTest() {
+	const server = await startServer(serverOptions)
+	const options = {
+		url: `http://localhost:${port}`,
+		method: 'GET',
+		tcp: true,
+	};
+	const result = await loadTest(options)
+	await server.close()
+	console.log(`Requests received: ${server.totalRequests}`)
+	result.show()
+}
+
+await runTcpPerformanceTest()
+

package/bin/testserver.js

@@ -5,49 +5,51 @@ import {startServer} from '../lib/testserver.js'
 import {loadConfig} from '../lib/config.js'
 import {getHalfCores, runTask} from '../lib/cluster.js'
 
+const configuration = loadConfig()
 const options = readOptions()
 start(options)
 
 
 function readOptions() {
 	const options = stdio.getopt({
+		port: {key: 'p', args: 1, description: 'Port for the server'},
 		delay: {key: 'd', args: 1, description: 'Delay the response for the given milliseconds'},
 		error: {key: 'e', args: 1, description: 'Return an HTTP error code'},
-		percent: {key: 'p', args: 1, description: 'Return an error (default 500) only for some % of requests'},
-		cores: {key: 'c', args: 1, description: 'Number of cores to use, default is half the total', default: getHalfCores()}
+		percent: {key: 'P', args: 1, description: 'Return an error (default 500) only for some % of requests'},
+		cores: {key: 'c', args: 1, description: 'Number of cores to use, default is half the total', default: getHalfCores()},
+		body: {key: 'b', args: 1, description: 'Body to return, default "OK"'},
+		file: {key: 'f', args: 1, description: 'File to read and return as body'},
 	});
-	const configuration = loadConfig()
 	if (options.args && options.args.length == 1) {
-		options.port = parseInt(options.args[0], 10);
-		if (!options.port) {
-			console.error('Invalid port');
-			options.printHelp();
-			process.exit(1);
-		}
+		options.port = options.port || options.args[0]
 	}
-	if(options.delay) {
-		if(isNaN(options.delay)) {
-			console.error('Invalid delay');
-			options.printHelp();
-			process.exit(1);
-		}
-		options.delay = parseInt(options.delay, 10);
+	return {
+		port: readInt(options, 'port'),
+		delay: readInt(options, 'delay'),
+		error: readInt(options, 'error'),
+		percent: readInt(options, 'percent'),
+		cores: readInt(options, 'cores'),
+		body: readString(options, 'body'),
+		file: readString(options, 'file'),
 	}
+}
 
-	if(!options.delay) {
-		options.delay = configuration.delay
-	}
-	if(!options.error) {
-		options.error = configuration.error
-	}
-	if(!options.percent) {
-		options.percent = configuration.percent
+function readString(options, key) {
+	return options[key] || configuration[key]
+}
+
+function readInt(options, key) {
+	if (options[key] && isNaN(options[key])) {
+		console.error(`Invalid ${key}`);
+		options.printHelp();
+		process.exit(1);
 	}
-	return options
+	const value = readString(options, key)
+	return parseInt(value) || undefined
 }
 
 function start(options) {
-	runTask(options.cores, async () => await startServer(options))
+	runTask(options.cores, async () => {await startServer(options)})
 }
 
 

package/doc/api.md

@@ -37,7 +37,7 @@ result.show()
 console.log('Tests run successfully')
 ```
 
-The call returns a `Result` object that contains all info about the load test, also described below.
+The call returns a `Result` object that contains all info about the load test, also described [below](#result).
 Call `result.show()` to display the results in the standard format on the console.
 
 As a legacy from before promises existed,
@@ -61,46 +61,6 @@ loadTest(options, function(error, result) {
 })
 ```
 
-
-Beware: if there are no `maxRequests` and no `maxSeconds`, then tests will run forever
-and will not call the callback.
-
-### Result
-
-The latency result returned at the end of the load test contains a full set of data, including:
-mean latency, number of errors and percentiles.
-A simplified example follows:
-
-```javascript
-{
-  url: 'http://localhost:80/',
-  maxRequests: 1000,
-  maxSeconds: 0,
-  concurrency: 10,
-  agent: 'none',
-  requestsPerSecond: undefined,
-  totalRequests: 1000,
-  percentiles: {
-	'50': 7,
-	'90': 10,
-	'95': 11,
-	'99': 15
-  },
-  effectiveRps: 2824,
-  elapsedSeconds: 0.354108,
-  meanLatencyMs: 7.72,
-  maxLatencyMs: 20,
-  totalErrors: 3,
-  errorCodes: {
-	'0': 1,
-	'500': 2
-  },
-}
-```
-
-The `result` object also has a `result.show()` function
-that displays the results on the console in the standard format.
-
 ### Options
 
 All options but `url` are, as their name implies, optional.
@@ -110,23 +70,34 @@ See also the [simplified list](../README.md#loadtest-parameters).
 
 The URL to invoke. Mandatory.
 
-#### `concurrency`
+#### `maxSeconds`
 
-How many clients to start in parallel.
+Max number of seconds to run the tests.
+Default is 10 seconds, applies only if no `maxRequests` is specified.
+
+Note: after the given number of seconds `loadtest` will stop sending requests,
+but may continue receiving tests afterwards.
+
+**Warning**: max seconds used to have no default value,
+so tests would run indefinitely if no `maxSeconds` and no `maxRequests` were specified.
+Max seconds was changed to default to 10 in version 8.
 
 #### `maxRequests`
 
 A max number of requests; after they are reached the test will end.
+Default is no limit;
+will keep on sending until the time limit in `maxSeconds` is reached.
 
 Note: the actual number of requests sent can be bigger if there is a concurrency level;
 loadtest will report just on the max number of requests.
 
-#### `maxSeconds`
+#### `concurrency`
 
-Max number of seconds to run the tests.
+How many clients to start in parallel, default is 10.
+Does not apply if `requestsPerSecond` is specified.
 
-Note: after the given number of seconds `loadtest` will stop sending requests,
-but may continue receiving tests afterwards.
+**Warning**: concurrency used to have a default value of 1,
+until it was changed to 10 in version 8.
 
 #### `timeout`
 
@@ -134,7 +105,7 @@ Timeout for each generated request in milliseconds. Setting this to 0 disables t
 
 #### `cookies`
 
-An array of cookies to send. Each cookie should be a string of the form name=value.
+An array of cookies to send. Each cookie should be a string of the form `name=value`.
 
 #### `headers`
 
@@ -146,9 +117,6 @@ like this:
         accept: "text/plain;text/html"
     }
 
-Note: when using the API, the "host" header is not inferred from the URL but needs to be sent
-explicitly.
-
 #### `method`
 
 The method to use: POST, PUT. Default: GET.
@@ -317,6 +285,86 @@ function contentInspector(result) {
     }
 },
 ```
+
+#### `tcp`
+
+If true, use low-level TCP sockets.
+Faster option that can increase performance by up to 10x,
+especially in local test setups.
+
+**Warning**: Experimental option.
+May not work for your test case.
+Not compatible with options `indexParam`, `statusCallback`, `requestGenerator`.
+See [TCP Sockets Performance](doc/tcp-sockets.md) for details.
+
+### Result
+
+The latency result returned at the end of the load test contains a full set of data, including:
+mean latency, number of errors and percentiles.
+A simplified example follows:
+
+```javascript
+{
+  url: 'http://localhost:80/',
+  maxRequests: 1000,
+  maxSeconds: 0,
+  concurrency: 10,
+  agent: 'none',
+  requestsPerSecond: undefined,
+  totalRequests: 1000,
+  percentiles: {
+	'50': 7,
+	'90': 10,
+	'95': 11,
+	'99': 15
+  },
+  effectiveRps: 2824,
+  elapsedSeconds: 0.354108,
+  meanLatencyMs: 7.72,
+  maxLatencyMs: 20,
+  totalErrors: 3,
+  clients: 10,
+  errorCodes: {
+	'0': 1,
+	'500': 2
+  },
+}
+```
+
+The `result` object also has a `result.show()` function
+that displays the results on the console in the standard format.
+
+Some of the attributes (`url`, `concurrency`) will be identical to the parameters passed.
+The following attributes can also be returned.
+
+#### `totalRequests`
+
+How many requests were actually processed.
+
+#### `totalRequests`
+
+How many requests resulted in an error.
+
+#### `effectiveRps`
+
+How many requests per second were actually processed.
+
+#### `elapsedSeconds`
+
+How many seconds the test lasted.
+
+#### `meanLatencyMs`
+
+Average latency in milliseconds.
+
+#### `errorCodes`
+
+Object containing a map with all status codes received.
+
+#### `clients`
+
+Number of concurrent clients started.
+Should equal the concurrency level unless the `rps` option is specified.
     
 ### Start Test Server
 
@@ -330,7 +378,7 @@ await server.close()
 ```
 
 This function returns when the server is up and running,
-with an HTTP server which can be `close()`d when it is no longer useful.
+with a server object which can be `close()`d when it is no longer useful.
 As a legacy from before promises existed,
 if an optional callback is passed as second parameter then it will not behave as `async`:
 
@@ -338,6 +386,9 @@ if an optional callback is passed as second parameter then it will not behave as
 const server = startServer({port: 8000}, error => console.error(error))
 ```
 
+**Warning**: up until version 7 this function returned an HTTP server;
+this was changed to a test server object with an identical `close()` method.
+
 The following options are available.
 
 #### `port`