drnbench 1.0.0

data/README.md

@@ -0,0 +1,296 @@
+# Drnbench
+
+## Description
+
+Drnbench is a benchmark tool for Droonga.
+
+It may be used for other HTTP servers.
+
+Drnbench provides features to send multiple random requests with different settings periodically.
+Number of clients (requests) in each period will be automatically increased gradually.
+So you'll be able to guess the limit performance of the throughput of a server, via the report like following:
+
+    n_clients,total_n_requests,queries_per_second,min_elapsed_time,max_elapsed_time,average_elapsed_time,200
+    1,33,3.3,0.164632187,0.164632187,0.19133309036363635,0
+    2,70,7.0,0.161510877,0.161510877,0.1846983412285715,0
+    3,87,8.7,0.1658357,0.1658357,0.24303329366666668,0
+    ...
+
+Drnbench provides two commands, for different styles of benchmarked servers.
+
+ * `drnbench-request-response`: benchmarks simple request-response style APIs.
+ * `drnbench-publish-subscribe`: benchmarks publish-subscribe style (including HTTP streaming) APIs.
+   This command is not implemented yet.
+
+## How to run benchmark?
+
+### Benchmarking with an HTTP server
+
+Drnbench can benchmark throughput performance of an HTTP server with random requests.
+
+In this scenario, you have to do:
+
+ * setup an HTTP server.
+ * prepare patterns of requests.
+
+Drnbench will start multiple clients and send many requests based on the patterns file.
+
+ 1. Create a patterns file in the format:
+    
+        {
+          "(pattern type 1 name)": {
+            "frequency": (appearance ratio in all requests),
+            "path":      "(path to the endpoint)",
+            "method":    "(HTTP method)",
+            "patterns":  [
+              { "body": (request body 1 sent by POST method) },
+              { "body": (request body 2 sent by POST method) },
+              ...
+            ]
+          }
+          "(patterns type 2 name)": {
+            "frequency": (appearance ratio in all requests),
+            "patterns":  [
+              {
+                "path":   "(path to the endpoint 1)",
+                "method": "(HTTP method)",
+                "body":   (request body 1 sent by POST method)
+              },
+              {
+                "path":   "(path to the endpoint 2)",
+                "method": "(HTTP method)",
+                "body":   (request body 2 sent by POST method)
+              },
+              ...
+            ]
+          },
+          ...
+        }
+    
+    For example, a file "patterns.json" like:
+    
+        {
+          "user search": {
+            "frequency": 0.61,
+            "method":    "GET",
+            "patterns":  [
+              { "path": "/users?q=foo" },
+              { "path": "/users?q=bar" },
+              ...
+            ]
+          },
+          "item search": {
+            "frequency": 0.32,
+            "method":    "GET",
+            "patterns":  [
+              { "path": "/items?q=foo" },
+              { "path": "/items?q=bar" },
+              ...
+            ]
+          },
+          ...
+        }
+    
+ 2. Setup an HTTP server. For example, localhost:80.
+ 3. Run drnbench with the pattern file.
+    
+        # cd ~/drnbench
+        # bundle exec bin/drnbench-request-response \
+            --start-n-clients=1 \
+            --end-n-clients=32 \
+            --step=1 \
+            --duration=10 \
+            --wait=0.01 \
+            --mode=http \
+            --request-patterns-file=/tmp/patterns.json \
+            --host=localhost \
+            --port=80
+    
+ 4. You'll get a report.
+
+
+### Benchmarking of request-responsne style commands, with a Droonga-based search system
+
+Drnbench can benchmark throughput performance of a Droonga-based search system with random requests.
+
+In this scenario, you have to do:
+
+ * setup a Droonga-based search system.
+ * prepare patterns of requests for commands.
+
+Drnbench will start multiple clients and send many requests based on the patterns file.
+
+ 1. Create a patterns file in the format:
+    
+        {
+          "(pattern type 1 name)": {
+            "frequency": (appearance ratio in all requests),
+            "command":   "(command name)",
+            "patterns":  [
+              { command parameters 1 },
+              { command parameters 2 },
+              { command parameters 3 },
+              ...
+            ]
+          }
+          "(patterns type 2 name)": {
+            ...
+          },
+          ...
+        }
+    
+    For example, a file "patterns.json" like:
+    
+        {
+          "user search": {
+            "frequency": 0.61,
+            "command":   "search",
+            "patterns":  [
+              {
+                "queries": {
+                  "users": {
+                    "source":    "User",
+                    "condition": "age >= 10",
+                    "sortBy":    { "keys": ["-birthday"], "offset": 0, "limit": 100" },
+                    "output":    {
+                      "elements": [
+                        "count",
+                        "records"
+                      ],
+                      "attributes": ["_key", "name", "age", "birhtday"],
+                      "offset": 0,
+                      "limit":  100
+                    }
+                  }
+                }
+              },
+              ...
+            ]
+          },
+          "item search": {
+            "frequency": 0.32,
+            "command":   "search",
+            "patterns":  [
+              {
+                "queries": {
+                  "users": {
+                    "source":    "Item",
+                    "condition": "visible == true",
+                    "sortBy":    { "keys": ["title"], "offset": 0, "limit": 100" },
+                    "output":    {
+                      "elements": [
+                        "count",
+                        "records"
+                      ],
+                      "attributes": ["title", "price"],
+                      "offset": 0,
+                      "limit":  100
+                    }
+                  }
+                }
+              },
+              ...
+            ]
+          },
+          ...
+        }
+    
+ 2. Setup a Droonga Engine server. For example, localhost:23003.
+ 3. Setup a Protocol Adapter server. For example, localhost:3003.
+ 4. Run drnbench with the pattern file.
+    
+        # cd ~/drnbench
+        # bundle exec bin/drnbench-request-response \
+            --start-n-clients=1 \
+            --end-n-clients=32 \
+            --step=1 \
+            --duration=10 \
+            --wait=0.01 \
+            --mode=http-droonga \
+            --request-patterns-file=/tmp/patterns.json \
+            --host=localhost \
+            --port=3003
+    
+ 5. You'll get a report.
+
+
+### Benchmarking of HTTP streaming APIs, with a Droonga-based search system
+
+Drnbench can benchmark an HTTP streaming API based on a publish-subscribe command.
+
+In this scenario, you have to do:
+
+ * prepare configuration files "fluentd.conf" and "catalog.json" for a Droonga Engine.
+ * prepare an express application works as a Droonga Protocol Adapter.
+ * prepare pattern files for subscribe request and feeded data.
+
+Drnbench will run benchmark like:
+
+ 1. Prepare subscribers.
+ 2. Send "feed" messages to the Droonga Engine.
+    All subscribers will receive all published messages.
+ 3. Increase the number of subscribers.
+ 4. Repeat 2, 3, and 4.
+
+Steps to run:
+
+ 1. Create a patterns file for a subscribe request:
+    
+        {
+          "path":    "/path/to/endpoint",
+          "method":  "HTTP method",
+          "headers": {
+            "X-xxxx": "xxxxx",
+          },
+          "body":    (sent as the request body)
+        }
+    
+    For example, a file "watch-subscribe.json" like:
+    
+        {
+          "path": "/droonga-streaming/watch?condition=keyword"
+        }
+    
+ 2. Create a patterns file for messages to be feeded:
+    
+        {
+          "type":    "(message type)",
+          "dataset": "(dataset name)",
+          "body":    (message body)
+        }
+    
+    For example, a file "watch-feed.json" like:
+    
+        {
+          "type":    "watch.feed",
+          "dataset": "Watch",
+          "body":    {
+            "targets": {
+              "body": "a content including the keyword"
+            }
+          }
+        }
+    
+ 3. Run drnbench with the pattern file.
+    
+        # cd ~/drnbench
+        # bundle exec bin/drnbench-publish-subscribe \
+            --start-n-subscribers=1000 \
+            --n-publishings=1000 \
+            --n-steps=10 \
+            --timeout=5 \
+            --subscribe-request-file=/tmp/watch-subscribe.json \
+            --feed-file=/tmp/watch-feed.json \
+            --protocol-adapter-port=80 \
+            --engine-config-path=/tmp/engine/
+    
+ 4. You'll get a report.
+
+
+## License
+
+GPLv3 or later.
+
+Copyright (c) 2013-2014 Droonga Project
+
+See LICENSE.txt for details.

data/Rakefile

@@ -0,0 +1,33 @@
+# -*- mode: ruby -*-
+#
+# Copyright (C) 2013-2014  Droonga Project
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+require "bundler/gem_helper"
+require "packnga"
+
+base_dir = File.join(File.dirname(__FILE__))
+helper = Bundler::GemHelper.new(base_dir)
+def helper.version_tag
+  version
+end
+
+helper.install
+spec = helper.gemspec
+
+Packnga::DocumentTask.new(spec) do |task|
+  task.original_language = "en"
+  task.translate_languages = ["ja"]
+end

data/bin/drnbench-publish-subscribe

@@ -0,0 +1,104 @@
+#!/usr/bin/env ruby
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2013-2014  Droonga Project
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+require "drnbench"
+require "optparse"
+require "shellwords"
+
+config = Drnbench::PublishSubscribe::Configuration.new
+option_parser = OptionParser.new do |parser|
+  parser.on("--start-n-subscribers=N", Integer,
+            "initial number of subscribers") do |n_subscribers|
+    config.start_n_subscribers = n_subscribers
+  end
+  parser.on("--n-publishings=N", Integer,
+            "number of publish times") do |n_publishings|
+    config.n_publishings = n_publishings
+  end
+  parser.on("--n-steps=N", Integer,
+            "number of benchmark steps") do |n_steps|
+    config.n_steps = n_steps
+  end
+  parser.on("--timeout=SECONDS", Float,
+            "timeout for receiving") do |timeout|
+    config.timeout = timeout
+  end
+  parser.on("--output-path=PATH", String,
+            "path to the output CSV file") do |output_path|
+    config.output_path = output_path
+  end
+
+  parser.on("--subscribe-request-file=PATH", String,
+            "path to the file which defines a request to subscribe") do |path|
+    config.subscribe_request_file = path
+  end
+  parser.on("--feed-file=PATH", String,
+            "path to the file which defines a message feeded to the engine") do |path|
+    config.feed_file = path
+  end
+
+  parser.on("--protocol-adapter-port=PORT", Integer,
+            "port number for the Droonga Protocol Adapter which is used for clients") do |port|
+    config.protocol_adapter.port = port
+  end
+  parser.on("--protocol-adapter-receive-port=PORT", Integer,
+            "port number for the Droonga Protocol Adapter which is used for the engine") do |port|
+    config.protocol_adapter.receive_port = port
+  end
+  parser.on("--protocol-adapter-application-dir=PATH", String,
+            "path to the directory of the Droonga Protocol Adapter") do |path|
+    config.protocol_adapter.application_dir = path
+  end
+  parser.on("--node=PATH", String,
+            "path to the node.js executable") do |node|
+    config.protocol_adapter.node = node
+  end
+  parser.on("--node-options=OPTIONS",
+            "options for node.js",
+            "you can specify this option multiple times") do |options|
+    config.protocol_adapter.node_options = Shellwords.split(options)
+  end
+
+  parser.on("--engine-config-path=PATH", String,
+            "path to the configuration directory for Droonga Engine") do |path|
+    config.engine.engine_config_path = path
+  end
+  parser.on("--fluentd=PATH", String,
+            "path to the fluentd executable") do |fluentd|
+    config.engine.fluentd = fluentd
+  end
+  parser.on("--fluentd-options=OPTIONS",
+            "options for fluentd",
+            "you can specify this option multiple times") do |options|
+    config.engine.fluentd_options = Shellwords.split(options)
+  end
+end
+args = option_parser.parse!(ARGV)
+
+config.validate
+
+runner = Drnbench::PublishSubscribe::GradualRunner.new(config)
+runner.run
+
+File.open(config.output_path, "w") do |file|
+  runner.total_results.each do |row|
+    file.puts(CSV.generate_line(row))
+    puts row.join(",")
+  end
+end
+puts "Statistics has been saved to #{config.output_path}"

data/bin/drnbench-request-response

@@ -0,0 +1,89 @@
+#!/usr/bin/env ruby
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2013-2014  Droonga Project
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+require "drnbench"
+require "optparse"
+
+config = Drnbench::RequestResponse::Configuration.new
+option_parser = OptionParser.new do |parser|
+  parser.on("--duration=SECONDS", Float,
+            "duration of the benmark") do |duration|
+    config.duration = duration
+  end
+  parser.on("--wait=SECONDS", Float,
+            "wait for each request") do |wait|
+    config.wait = wait
+  end
+  parser.on("--start-n-clients=N", Integer,
+            "initial number of clients (optional)") do |n_clients|
+    config.start_n_clients = n_clients
+  end
+  parser.on("--end-n-clients=N", Integer,
+            "final number of clients (optional)") do |n_clients|
+    config.end_n_clients = n_clients
+  end
+  parser.on("--step=COUNT", Integer,
+            "step to increase number of clients (optional)") do |step|
+    config.step = step
+  end
+
+  parser.on("--mode=MODE", String,
+            "mode of benchmark (optional)",
+            "available modes:",
+            "  http",
+            "  http-droonga-search") do |mode|
+    config.mode = mode.gsub(/-/, "_")
+  end
+  parser.on("--request-patterns-file=PATH",
+            "path to request patterns JSON file") do |path|
+    config.request_patterns_file = File.expand_path(path)
+  end
+
+  parser.on("--default-host=HOST", String,
+            "default host name (optional)") do |host|
+    config.default_host = host
+  end
+  parser.on("--default-port=PORT", Integer,
+            "default port number (optional)") do |port|
+    config.default_port = port
+  end
+  parser.on("--default-path=PATH", String,
+            "default path (optional)") do |path|
+    config.default_path = path
+  end
+  parser.on("--default-method=METHOD", String,
+            "default HTTP method (optional)") do |method|
+    config.default_method = method
+  end
+
+  parser.on("--output-path=PATH",
+            "path to output statistics as a CSV file (optional)") do |path|
+    config.output_path = File.expand_path(path)
+  end
+end
+args = option_parser.parse!(ARGV)
+
+config.validate
+
+runner = Drnbench::RequestResponse::GradualRunner.new(config)
+runner.run
+
+File.open(config.output_path, "w") do |file|
+  file.puts runner.result.to_csv
+end
+puts "Statistics has been saved to #{config.output_path}"