logstash-perftool 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: e6022c1e0ceb936a26ecfa1d2e3523de3d24e836
+  data.tar.gz: 30e95250b7a8377d36cd8e3cae983795d7262bec
+SHA512:
+  metadata.gz: 62171012bd2ba7bd5dd46e3bf271a43a2ca96ef6c7390884465262cef9f17efaad3a94e42e45d178318bbdf6abc43154f25f1990455916a400c5f7eb99dae62f
+  data.tar.gz: 613962482e0a065a643ce9c9775f2e143433e812ce2947eb4c1699e21f91ab6db6b1a331e512bbd7082074c49d17dceeb98ee5621e9b6b055d1a48b575e36d25

data/.gitignore

@@ -0,0 +1,4 @@
+*.lock
+*.gem
+config.txt
+workspace/

data/.rspec

@@ -0,0 +1,3 @@
+--format documentation
+--colour
+--tty

data/CONTRIBUTORS

@@ -0,0 +1,11 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* Pere Urbon-Bayes (purbon)
+* Colin Surprenant (colin)
+
+Note: If you've sent me patches, bug reports, or otherwise contributed to
+logstash, and you aren't on the list above and want to be, please let me know
+and I'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/Gemfile

@@ -0,0 +1,3 @@
+source 'http://rubygems.org'
+
+gemspec

data/LICENSE

@@ -0,0 +1,13 @@
+Copyright (c) 2012–2015 Elasticsearch <http://www.elastic.co>
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/README.md

@@ -0,0 +1,119 @@
+# Performance Testing for Logstash
+
+## Installation
+
+You can use this code as a gem within your logstash project, to proceed with the installation you can either download the code and build the gem using the next command:
+
+To run a benchmark using the Logstash Performance meter tool you will need to install this gem in your system, and to do it you can run the next command:
+
+* ```gem install logstash-perftool```
+
+This will make the last version of this gem available to you.
+
+or, if you like to be on the edge, you can add it to your Gemfile like this:
+
+    gem 'logstash-perftool', :git => 'https://github.com/elastic/logstash-performance-testing.git'
+
+and then do budler update.
+
+## Setup and Runtime
+
+The most simple scenario you could find is using the default set of
+test, available in this gem. To do this you can simple run the ```lsperfm```
+from the root of your Logstash installation.
+
+If you like to add you own configurations and test suites,  you need the next data (you can see an example of them at the `examples/` directory):
+
+- The logstash configs, found in `..config/`
+- The sample input files, found in `..input/`
+- The suites definitions, found in `..suite/`
+
+### Configuration
+
+If you add a file named ```.lsperfm.yml``` in your main logstash directory you can have your configuration and input files in non standard
+location.
+
+Example:
+
+```
+default:
+  path: 'config-path'
+  config: ''
+  input: ''
+```
+
+### Bootstrap
+
+Before you can run your test is necessary to bootstrap your logstash installation and install the test dependencies, to do that you must:
+
+If you are in 1.5.x:
+- Run `rake bootstrap` to setup the system.
+- Run `lsperfm-deps` to install the test dependencies
+For 1.4:
+- Run `bin/logstash deps` to setup everything.
+
+## Performance tests
+
+The test are run in groups called suites.
+
+### How to execute the default tests
+
+This is the most simple use case you can have. To run the default tests
+you can simply run ```lsperfm``` from the root of your Logstash
+installation and the tool will use the default test suite.
+
+### How to run a custom test suite
+
+- suites examples can be found in `examples/suite/`
+
+```
+lsperfm [suite definition]
+```
+
+a suite file defines a series of tests to run.
+
+#### suite file format
+
+```ruby
+# each test can be executed by either target duration using :time => N secs
+# or by number of events with :events => N
+#
+#[
+#  {:name => "simple json out", :config => "config/simple_json_out.conf", :input => "input/simple_10.txt", :time => 30},
+#  {:name => "simple json out", :config => "config/simple_json_out.conf", :input => "input/simple_10.txt", :events => 50000},
+#]
+#
+[
+  {:name => "simple json out", :config => "config/simple_json_out.conf", :input => "input/simple_10.txt", :time => 60},
+  {:name => "simple line out", :config => "config/simple.conf", :input => "input/simple_10.txt", :time => 60},
+  {:name => "json codec", :config => "config/json_inout_codec.conf", :input => "input/json_medium.txt", :time => 60},
+  {:name => "json filter", :config => "config/json_inout_filter.conf", :input => "input/json_medium.txt", :time => 60},
+  {:name => "complex syslog", :config => "config/complex_syslog.conf", :input => "input/syslog_acl_10.txt", :time => 60},
+]
+```
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports,
+complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and
+maintainers or community members  saying "send patches or die" - you will not
+see that here.
+
+It is more important to me that you are able to contribute.
+
+### Contribution Steps
+
+1. Test your changes! Write test and run the test suites.
+2. Please make sure you have signed our [Contributor License
+   Agreement](http://www.elastic.co/contributor-agreement/). We are not
+   asking you to assign copyright to us, but to give us the right to distribute
+   your code without restriction. We ask this of all contributors in order to
+   assure our users of the origin and continuing existence of the code. You
+   only need to sign the CLA once.
+3. Send a pull request! Push your changes to your fork of the repository and
+   [submit a pull
+   request](https://help.github.com/articles/using-pull-requests). In the pull
+   request, describe what your changes do and mention any bugs/issues related
+   to the pull request.

data/Rakefile

@@ -0,0 +1,10 @@
+require "bundler/gem_tasks"
+
+require 'rspec'
+require 'rspec/core/rake_task'
+
+desc "Run all specs"
+RSpec::Core::RakeTask.new(:spec) do |t|
+  t.fail_on_error = true
+  t.verbose = false
+end

data/bin/lsperfm

@@ -0,0 +1,9 @@
+#!/usr/bin/env ruby
+
+# encoding: utf-8
+
+$LOAD_PATH << "."
+
+require 'lsperfm'
+
+LogStash::PerformanceMeter.invoke

data/bin/lsperfm-deps

@@ -0,0 +1,32 @@
+#!/usr/bin/env ruby
+
+# encoding: utf-8
+
+puts "installing dependencies..."
+
+logstash_home = ENV['LOGSTASH_HOME'] || Dir.pwd
+base_dir = (ARGV.size < 1 ?  logstash_home : ARGV[0])
+
+logstash = File.join(base_dir, "bin", "logstash")
+version = `#{logstash} --version`
+
+if version[/\d\.\d\.\d/] =~ /1\.4\./
+  # why do deps here? can't we assume you must have a working logstash distribution?
+  # `#{logstash} deps 2>&1`
+else
+  inputs  = ['stdin'].map{|s| "input-#{s}"}
+  outputs = ['stdout'].map{|s| "output-#{s}"}
+  filters = ['clone', 'json', 'grok', 'syslog_pri', 'date', 'mutate'].map{|s| "filter-#{s}"}
+
+  # why do bootstrap here? can't we assume you must have a working logstash distribution?
+  # `#{rake} bootstrap`
+
+  [inputs, outputs, filters].each do |plugins|
+    plugins.map{|s| "logstash-#{s}"}.each do |plugin|
+      command = "#{File.join(base_dir, "bin", "plugin")} install #{plugin}"
+      puts "#{command}\n#{%x[#{command}]}"
+    end
+  end
+end
+
+puts "done!"

data/examples/config/complex_syslog.conf

@@ -0,0 +1,46 @@
+input {
+  stdin {
+    type => syslog
+  }
+}
+
+filter {
+  if [type] == "syslog" {
+    grok {
+      match => { "message" => "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{PROG:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
+      add_field => [ "received_at", "%{@timestamp}" ]
+      add_field => [ "received_from", "%{syslog_hostname}" ]
+    }
+    syslog_pri { }
+    date {
+      match => ["syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
+    }
+
+    if [syslog_timestamp] {
+      mutate {
+        add_field => [ "[times][created_at]",  "%{syslog_timestamp}"]
+        add_field => [ "[times][received_at]",  "%{@timestamp}"]
+      }
+    }
+
+    mutate {
+      add_field => [ "[hosts][source]",  "%{received_from}"]
+      add_field => [ "[level][facility]",  "%{syslog_facility}"]
+      add_field => [ "[level][severity]",  "%{syslog_severity}"]
+    }
+
+    if !("_grokparsefailure" in [tags]) {
+      mutate {
+        replace => [ "@source_host", "%{syslog_hostname}" ]
+        replace => [ "@message", "%{syslog_message}" ]
+      }
+    }
+    mutate {
+      remove_field => [ "syslog_hostname", "syslog_message", "syslog_timestamp" ]
+    }
+  }
+}
+
+output {
+  stdout { codec => json_lines }
+}

data/examples/config/json_inout_codec.conf

@@ -0,0 +1,11 @@
+input {
+  stdin { codec => "json_lines" }
+}
+
+filter {
+  clone {}
+}
+
+output {
+  stdout { codec => json_lines }
+}

data/examples/config/json_inout_filter.conf

@@ -0,0 +1,11 @@
+input {
+  stdin {}
+}
+
+filter {
+  json { source => "message" }
+}
+
+output {
+  stdout { codec => json_lines }
+}

data/examples/config/simple.conf

@@ -0,0 +1,11 @@
+input {
+  stdin {}
+}
+
+filter {
+  clone {}
+}
+
+output {
+  stdout { codec => line }
+}

data/examples/config/simple_grok.conf

@@ -0,0 +1,13 @@
+input {
+	stdin { type => "apache" }
+}
+
+filter {
+	grok {
+		match => {"message" => "%{COMBINEDAPACHELOG}"}
+	}
+}
+
+output {
+	stdout { codec => line }
+}

data/examples/config/simple_json_out.conf

@@ -0,0 +1,11 @@
+input {
+  stdin {}
+}
+
+filter {
+  clone {}
+}
+
+output {
+  stdout { codec => json_lines }
+}

data/examples/input/apache_log.txt

@@ -0,0 +1,30 @@
+83.149.9.216 - - [17/Sep/2014:07:13:42 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [17/Sep/2014:07:13:42 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-dashboard3.png HTTP/1.1" 200 171717 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [17/Sep/2014:07:13:44 +0000] "GET /presentations/logstash-monitorama-2013/plugin/highlight/highlight.js HTTP/1.1" 200 26185 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [17/Sep/2014:07:13:44 +0000] "GET /presentations/logstash-monitorama-2013/plugin/zoom-js/zoom.js HTTP/1.1" 200 7697 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [17/Sep/2014:07:13:45 +0000] "GET /presentations/logstash-monitorama-2013/plugin/notes/notes.js HTTP/1.1" 200 2892 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [17/Sep/2014:07:13:42 +0000] "GET /presentations/logstash-monitorama-2013/images/sad-medic.png HTTP/1.1" 200 430406 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [17/Sep/2014:07:13:45 +0000] "GET /presentations/logstash-monitorama-2013/css/fonts/Roboto-Bold.ttf HTTP/1.1" 200 38720 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [17/Sep/2014:07:13:45 +0000] "GET /presentations/logstash-monitorama-2013/css/fonts/Roboto-Regular.ttf HTTP/1.1" 200 41820 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [17/Sep/2014:07:13:45 +0000] "GET /presentations/logstash-monitorama-2013/images/frontend-response-codes.png HTTP/1.1" 200 52878 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [17/Sep/2014:07:13:43 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-dashboard.png HTTP/1.1" 200 321631 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [17/Sep/2014:07:13:46 +0000] "GET /presentations/logstash-monitorama-2013/images/Dreamhost_logo.svg HTTP/1.1" 200 2126 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [17/Sep/2014:07:13:43 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-dashboard2.png HTTP/1.1" 200 394967 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [17/Sep/2014:07:13:46 +0000] "GET /presentations/logstash-monitorama-2013/images/apache-icon.gif HTTP/1.1" 200 8095 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [17/Sep/2014:07:13:46 +0000] "GET /presentations/logstash-monitorama-2013/images/nagios-sms5.png HTTP/1.1" 200 78075 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [17/Sep/2014:07:13:46 +0000] "GET /presentations/logstash-monitorama-2013/images/redis.png HTTP/1.1" 200 25230 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [17/Sep/2014:07:13:47 +0000] "GET /presentations/logstash-monitorama-2013/images/elasticsearch.png HTTP/1.1" 200 8026 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [17/Sep/2014:07:13:47 +0000] "GET /presentations/logstash-monitorama-2013/images/logstashbook.png HTTP/1.1" 200 54662 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [17/Sep/2014:07:13:47 +0000] "GET /presentations/logstash-monitorama-2013/images/github-contributions.png HTTP/1.1" 200 34245 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [17/Sep/2014:07:13:47 +0000] "GET /presentations/logstash-monitorama-2013/css/print/paper.css HTTP/1.1" 200 4254 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [17/Sep/2014:07:13:47 +0000] "GET /presentations/logstash-monitorama-2013/images/1983_delorean_dmc-12-pic-38289.jpeg HTTP/1.1" 200 220562 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [17/Sep/2014:07:13:46 +0000] "GET /presentations/logstash-monitorama-2013/images/simple-inputs-filters-outputs.jpg HTTP/1.1" 200 1168622 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [17/Sep/2014:07:13:46 +0000] "GET /presentations/logstash-monitorama-2013/images/tiered-outputs-to-inputs.jpg HTTP/1.1" 200 1079983 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [17/Sep/2014:07:13:53 +0000] "GET /favicon.ico HTTP/1.1" 200 3638 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+24.236.252.67 - - [17/Sep/2014:07:14:10 +0000] "GET /favicon.ico HTTP/1.1" 200 3638 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:26.0) Gecko/20100101 Firefox/26.0"
+93.114.45.13 - - [17/Sep/2014:07:14:32 +0000] "GET /articles/dynamic-dns-with-dhcp/ HTTP/1.1" 200 18848 "http://www.google.ro/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&ved=0CCwQFjAB&url=http%3A%2F%2Fwww.semicomplete.com%2Farticles%2Fdynamic-dns-with-dhcp%2F&ei=W88AU4n9HOq60QXbv4GwBg&usg=AFQjCNEF1X4Rs52UYQyLiySTQxa97ozM4g&bvm=bv.61535280,d.d2k" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"
+93.114.45.13 - - [17/Sep/2014:07:14:32 +0000] "GET /reset.css HTTP/1.1" 200 1015 "http://www.semicomplete.com/articles/dynamic-dns-with-dhcp/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"
+93.114.45.13 - - [17/Sep/2014:07:14:33 +0000] "GET /style2.css HTTP/1.1" 200 4877 "http://www.semicomplete.com/articles/dynamic-dns-with-dhcp/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"
+93.114.45.13 - - [17/Sep/2014:07:14:33 +0000] "GET /favicon.ico HTTP/1.1" 200 3638 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"
+93.114.45.13 - - [17/Sep/2014:07:14:33 +0000] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://www.semicomplete.com/articles/dynamic-dns-with-dhcp/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"
+93.114.45.13 - - [17/Sep/2014:07:14:33 +0000] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/style2.css" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"

data/examples/input/json_medium.txt

@@ -0,0 +1,10 @@
+{"_scroll_id":"xxx", "took":5, "timed_out":false, "_shards":{"total":15,"successful":15,"failed":0}, "hits":{"total":1000050, "max_score":1.0, "hits":[{"_index":"logstash2", "_type":"logs", "_id":"AmaqL7VuSWKF-F6N_Gz72g", "_score":1.0, "_source" : {"message":"foobar", "@version":"1", "@timestamp":"2014-05-19T21:08:39.000Z", "host":"colin-mbp13r"} } ] } }
+{"_scroll_id":"xxx", "took":5, "timed_out":false, "_shards":{"total":15,"successful":15,"failed":0}, "hits":{"total":1000050, "max_score":1.0, "hits":[{"_index":"logstash2", "_type":"logs", "_id":"AmaqL7VuSWKF-F6N_Gz72g", "_score":1.0, "_source" : {"message":"foobar", "@version":"1", "@timestamp":"2014-05-19T21:08:39.000Z", "host":"colin-mbp13r"} } ] } }
+{"_scroll_id":"xxx", "took":5, "timed_out":false, "_shards":{"total":15,"successful":15,"failed":0}, "hits":{"total":1000050, "max_score":1.0, "hits":[{"_index":"logstash2", "_type":"logs", "_id":"AmaqL7VuSWKF-F6N_Gz72g", "_score":1.0, "_source" : {"message":"foobar", "@version":"1", "@timestamp":"2014-05-19T21:08:39.000Z", "host":"colin-mbp13r"} } ] } }
+{"_scroll_id":"xxx", "took":5, "timed_out":false, "_shards":{"total":15,"successful":15,"failed":0}, "hits":{"total":1000050, "max_score":1.0, "hits":[{"_index":"logstash2", "_type":"logs", "_id":"AmaqL7VuSWKF-F6N_Gz72g", "_score":1.0, "_source" : {"message":"foobar", "@version":"1", "@timestamp":"2014-05-19T21:08:39.000Z", "host":"colin-mbp13r"} } ] } }
+{"_scroll_id":"xxx", "took":5, "timed_out":false, "_shards":{"total":15,"successful":15,"failed":0}, "hits":{"total":1000050, "max_score":1.0, "hits":[{"_index":"logstash2", "_type":"logs", "_id":"AmaqL7VuSWKF-F6N_Gz72g", "_score":1.0, "_source" : {"message":"foobar", "@version":"1", "@timestamp":"2014-05-19T21:08:39.000Z", "host":"colin-mbp13r"} } ] } }
+{"_scroll_id":"xxx", "took":5, "timed_out":false, "_shards":{"total":15,"successful":15,"failed":0}, "hits":{"total":1000050, "max_score":1.0, "hits":[{"_index":"logstash2", "_type":"logs", "_id":"AmaqL7VuSWKF-F6N_Gz72g", "_score":1.0, "_source" : {"message":"foobar", "@version":"1", "@timestamp":"2014-05-19T21:08:39.000Z", "host":"colin-mbp13r"} } ] } }
+{"_scroll_id":"xxx", "took":5, "timed_out":false, "_shards":{"total":15,"successful":15,"failed":0}, "hits":{"total":1000050, "max_score":1.0, "hits":[{"_index":"logstash2", "_type":"logs", "_id":"AmaqL7VuSWKF-F6N_Gz72g", "_score":1.0, "_source" : {"message":"foobar", "@version":"1", "@timestamp":"2014-05-19T21:08:39.000Z", "host":"colin-mbp13r"} } ] } }
+{"_scroll_id":"xxx", "took":5, "timed_out":false, "_shards":{"total":15,"successful":15,"failed":0}, "hits":{"total":1000050, "max_score":1.0, "hits":[{"_index":"logstash2", "_type":"logs", "_id":"AmaqL7VuSWKF-F6N_Gz72g", "_score":1.0, "_source" : {"message":"foobar", "@version":"1", "@timestamp":"2014-05-19T21:08:39.000Z", "host":"colin-mbp13r"} } ] } }
+{"_scroll_id":"xxx", "took":5, "timed_out":false, "_shards":{"total":15,"successful":15,"failed":0}, "hits":{"total":1000050, "max_score":1.0, "hits":[{"_index":"logstash2", "_type":"logs", "_id":"AmaqL7VuSWKF-F6N_Gz72g", "_score":1.0, "_source" : {"message":"foobar", "@version":"1", "@timestamp":"2014-05-19T21:08:39.000Z", "host":"colin-mbp13r"} } ] } }
+{"_scroll_id":"xxx", "took":5, "timed_out":false, "_shards":{"total":15,"successful":15,"failed":0}, "hits":{"total":1000050, "max_score":1.0, "hits":[{"_index":"logstash2", "_type":"logs", "_id":"AmaqL7VuSWKF-F6N_Gz72g", "_score":1.0, "_source" : {"message":"foobar", "@version":"1", "@timestamp":"2014-05-19T21:08:39.000Z", "host":"colin-mbp13r"} } ] } }

data/examples/input/simple_10.txt

@@ -0,0 +1,10 @@
+test 01
+test 02
+test 03
+test 04
+test 05
+test 06
+test 07
+test 08
+test 09
+test 10

data/examples/input/syslog_acl_10.txt

@@ -0,0 +1,10 @@
+<164>Oct 26 15:19:25 1.2.3.4 %ASA-4-106023: Deny udp src DRAC:10.1.2.3/43434 dst outside:192.168.0.1/53 by access-group "acl_drac" [0x0, 0x0]
+<164>Oct  6 15:20:25 2.2.3.4 %ASA-4-106023: Deny udp src DRAC:10.1.2.4/43434 dst outside:192.168.0.1/53 by access-group "acl_drac" [0x0, 0x0]
+<164>Oct  1 15:21:25 3.2.3.4 %ASA-4-106023: Allow tcp src DRAC:10.1.2.5/43434 dst outside:192.168.0.1/53 by access-group "acl_drac" [0x0, 0x0]
+<164>Oct 30 15:22:25 4.2.3.4 %ASA-4-106023: Allow tcp src DRAC:10.1.2.6/43434 dst outside:192.168.0.1/53 by access-group "acl_drac" [0x0, 0x0]
+<164>Oct 26 15:19:25 1.2.3.4 %ASA-4-106023: Deny udp src DRAC:10.1.2.3/43434 dst outside:192.168.0.1/53 by access-group "acl_drac" [0x0, 0x0]
+<164>Oct  6 15:20:25 2.2.3.4 %ASA-4-106023: Deny udp src DRAC:10.1.2.4/43434 dst outside:192.168.0.1/53 by access-group "acl_drac" [0x0, 0x0]
+<164>Oct  1 15:21:25 3.2.3.4 %ASA-4-106023: Allow tcp src DRAC:10.1.2.5/43434 dst outside:192.168.0.1/53 by access-group "acl_drac" [0x0, 0x0]
+<164>Oct 30 15:22:25 4.2.3.4 %ASA-4-106023: Allow tcp src DRAC:10.1.2.6/43434 dst outside:192.168.0.1/53 by access-group "acl_drac" [0x0, 0x0]
+<164>Oct 26 15:19:25 1.2.3.4 %ASA-4-106023: Deny udp src DRAC:10.1.2.3/43434 dst outside:192.168.0.1/53 by access-group "acl_drac" [0x0, 0x0]
+<164>Oct  6 15:20:25 2.2.3.4 %ASA-4-106023: Deny udp src DRAC:10.1.2.4/43434 dst outside:192.168.0.1/53 by access-group "acl_drac" [0x0, 0x0]

data/examples/suite/basic_performance_long.rb

@@ -0,0 +1,18 @@
+# format description:
+# each test can be executed by either target duration using :time => N secs
+# or by number of events with :events => N
+#
+#[
+#  {:name => "simple json out", :config => "config/simple_json_out.conf", :input => "input/simple_10.txt", :time => 30},
+#  {:name => "simple json out", :config => "config/simple_json_out.conf", :input => "input/simple_10.txt", :events => 50000},
+#]
+#
+[
+  {:name => "simple line in/out", :config => "config/simple.conf", :input => "input/simple_10.txt", :time => 120},
+  {:name => "simple line in/json out", :config => "config/simple_json_out.conf", :input => "input/simple_10.txt", :time => 120},
+  {:name => "json codec in/out", :config => "config/json_inout_codec.conf", :input => "input/json_medium.txt", :time => 120},
+  {:name => "line in/json filter/json out", :config => "config/json_inout_filter.conf", :input => "input/json_medium.txt", :time => 120},
+  {:name => "apache in/json out", :config => "config/simple.conf", :input => "input/apache_log.txt", :time => 120},
+  {:name => "apache in/grok codec/json out", :config => "config/simple_grok.conf", :input => "input/apache_log.txt", :time => 120},
+  {:name => "syslog in/json out", :config => "config/complex_syslog.conf", :input => "input/syslog_acl_10.txt", :time => 120},
+]

data/examples/suite/basic_performance_quick.rb

@@ -0,0 +1,18 @@
+# format description:
+# each test can be executed by either target duration using :time => N secs
+# or by number of events with :events => N
+#
+#[
+#  {:name => "simple json out", :config => "config/simple_json_out.conf", :input => "input/simple_10.txt", :time => 30},
+#  {:name => "simple json out", :config => "config/simple_json_out.conf", :input => "input/simple_10.txt", :events => 50000},
+#]
+#
+[
+  {:name => "simple line in/out", :config => "config/simple.conf", :input => "input/simple_10.txt", :time => 30},
+  {:name => "simple line in/json out", :config => "config/simple_json_out.conf", :input => "input/simple_10.txt", :time => 30},
+  {:name => "json codec in/out", :config => "config/json_inout_codec.conf", :input => "input/json_medium.txt", :time => 30},
+  {:name => "line in/json filter/json out", :config => "config/json_inout_filter.conf", :input => "input/json_medium.txt", :time => 30},
+  {:name => "apache in/json out", :config => "config/simple.conf", :input => "input/apache_log.txt", :time => 30},
+  {:name => "apache in/grok codec/json out", :config => "config/simple_grok.conf", :input => "input/apache_log.txt", :time => 30},
+  {:name => "syslog in/json out", :config => "config/complex_syslog.conf", :input => "input/syslog_acl_10.txt", :time => 30},
+]