bigbench 0.0.1 → 0.0.2

data/README.textile

@@ -1,63 +1,77 @@
 h1. BigBench
 
-h2. Installation
+BigBench is a http penetration tool. It allows you to test the performance of any web server with very high loads.
+
+h3. Installation
 
 bc.. gem install bigbench
 
-h2. Requirements
+h3. Further Reading
+
+* "BigBench Website":http://southdesign.github.com/bigbench
+* "BigBench RDoc":http://southdesign.github.com/bigbench/doc/BigBench.html
+
+h3. Requirements
 
 * Ruby 1.9+
 * Redis - only if you're testing with multiple hosts
 
-h2. What is this?
 
-BigBench is a http penetration tool. It allows you to test the performance of any web server with very high loads.
 
-h2. How do the test receipts look like?
 
-As easy as possible. For example like this in @example.rb@:
+h2. Test Receipts
+
+How do the test receipts look like? As easy as possible. For example like this in @example.rb@:
 
 bc.. configure => {
     :duration   => 2.minutes,
     :output     => "example.ljson",
-    :threads    => 5
+    :users      => 5,
+    :basic_auth => ['username', 'password']
 }
 
 benchmark "default website pages" => "http://localhost:3000" do
     get "/"
     get "/blog"
     get "/imprint"
+    get "/admin", :basic_auth => ['username', 'password']
 end
 
 benchmark "login and logout" => "http://localhost:3000" do
-    post "/login", { :name => "test@user.com", :password => "secret" }
-    post "/logout"
+    post "/login",  :params => { :name => "test@user.com", :password => "secret" }
+    post "/logout", :params => { :name => "test@user.com" }
 end
 
-h2. Can I test with a single machine?
 
-Yes, BigBench allows you to run your tests against every host from your local machine. The command for this looks like this:
+
+
+h2. Single Host vs. Multiple Hosts Testing
+
+You can either test with a single machine right from your local host, or with multiple machines using bots. No matter what, the test receipt will stay the same.
+
+h3. Single Host
+
+BigBench allows you to run your tests against every host from your local machine. The command for this looks like this:
 
 bc.. bigbench run local example.rb
 
-h2. Can I also test on multiple machines?
+h3. Multiple Hosts with Bots
 
-Yes! BigBench uses a bot design pattern which means you can run your tests from multiple hosts. Everything you need for this is a redis that
-is reachable from all testing hosts. Every host simply starts a bot that is checking for a new test receipt every minute like this:
+BigBench uses a bot design pattern which means you can run your tests from multiple hosts. Everything you need for this is a redis that is reachable from all testing hosts. Every host simply starts a bot that is checking for a new test receipt every minute like this:
 
 bc.. bigbench start bot redis_url:port redis_password
 
-Then to run the tests from all hosts simply use the same receipt as you would use for a local run and call it like this:
+p. Then to run the tests from all hosts simply use the same receipt as you would use for a local run and call it like this:
 
 bc.. bigbench run bots example.rb redis_url:port redis_password
 
-This will upload the test receipt to all bots and make them run it. Every bot reports its results back to the redis and the local machine then
-combines, and writes them to the output file. So you test with the same receipts and get the same results, no matter if your testing from the local host
-or with multiple bots.
+p. This will upload the test receipt to all bots and make them run it. Every bot reports its results back to the redis and the local machine then combines, and writes them to the output file. So you test with the same receipts and get the same results, no matter if your testing from the local host or with multiple bots.
 
-h3. How does the recorded output look like?
 
-It's in the @*.ljson*@ format which is nothing else but a textfile with a complete JSON object on every line. It looks like this:
+
+h2. Output
+
+How does the recorded output look like? It's in the @*.ljson*@ format which is nothing else but a textfile with a complete JSON object on every line. It looks like this:
 
 bc.. {"elapsed":0.002233,"start":1333981203.542233,"stop":1333981203.54279,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
 {"elapsed":0.00331,"start":1333981203.5434968,"stop":1333981203.5438669,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
@@ -77,5 +91,29 @@ bc.. {"elapsed":0.002233,"start":1333981203.542233,"stop":1333981203.54279,"dura
 {"elapsed":0.01592,"start":1333981203.556119,"stop":1333981203.5564768,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
 ...
 
-p. The advantage with this file format is, that it can be parsed and computed very efficiently because the JSON parser doesn't have to parse a whole JSON array with
-with loads of objects but simply one object - line by line.
+p. The advantage with this file format is, that it can be parsed and computed very efficiently because the JSON parser doesn't have to parse a whole JSON array with with loads of objects but simply one object - line by line.
+
+
+
+h2. Load Comparison
+
+BigBench is awfully good at creating high loads on web servers. A quick benchmark comparison to "Apache's JMeter":http://jmeter.apache.org/ shows that BigBench is able to create 16% more load than JMeter.
+
+h3. Test Configuration: Apache's JMeter vs. BigBench
+
+|_. Parameter           |_. Value                               |
+| Test Duration         | 2 Minutes                             |
+| Concurrency(Threads)  | 20                                    |
+| Rack Server           | Thin                                  |
+| Rack Host             | localhost                             |
+| Rack Request          | GET: 200, Body: "Test"                |
+| Ruby Version          | ruby 1.9.3p125 [x86_64-darwin11.3.0]  |
+| JMeter Version        | 2.6 r1237317                          |
+| BigBench Version      | 0.2                                   |
+
+h3. Test Results
+
+|_. Value               |_. JMeter          |_. BigBench        |
+| Total Requests        | 48.014            | 55.484            |
+| Requests/sec          | 377               | 462               |
+| Percentages           | 100%%             | 116%              |

data/Rakefile

@@ -1,4 +1,5 @@
 require "bundler/gem_tasks"
+require "rack"
 
 namespace :bigbench do
   
@@ -7,4 +8,10 @@ namespace :bigbench do
     `rdoc --main BigBench --include lib/bigbench/help/ && open doc/index.html`
   end
   
+  desc "Start Thin RDoc Testserver"
+  task :testserver do
+    load "spec/lib/test_web_server.rb"
+    Rack::Handler::Thin.run Sinatra::Application, :Port => 3001
+  end
+  
 end

data/bigbench.gemspec

@@ -23,7 +23,12 @@ Gem::Specification.new do |s|
   s.add_development_dependency "active_support"
   s.add_development_dependency "rack"
   s.add_development_dependency "thin"
+  s.add_development_dependency "eventmachine"
+  s.add_development_dependency "em-http-request"
+  s.add_development_dependency "sinatra"
   
   s.add_runtime_dependency "redis"
   s.add_runtime_dependency "active_support"
+  s.add_runtime_dependency "eventmachine"
+  s.add_runtime_dependency "em-http-request"
 end

data/dev/net_http.rb

@@ -0,0 +1,78 @@
+require 'benchmark'
+require 'net/http'
+require 'eventmachine'
+require 'em-http'
+require 'active_support/all'
+
+{ :test => :a }.to_json
+
+n = 1_000
+uri = URI("http://localhost:3000/")
+
+Benchmark.bm do |benchmark|
+  
+  benchmark.report "single start block" do
+    
+    Net::HTTP.start(uri.host, uri.port) do |http|
+      0.upto(n) do
+        request = Net::HTTP::Get.new(uri.request_uri)
+        response = http.request(request)
+      end
+    end
+    
+  end
+  
+  benchmark.report "straight http object" do
+    
+    Net::HTTP.start(uri.host, uri.port) do |http|
+      0.upto(n) do
+        http.get(uri.request_uri)
+      end
+    end
+    
+  end
+  
+  benchmark.report "new start block" do
+    
+    0.upto(n) do
+      Net::HTTP.start(uri.host, uri.port) do |http|
+        request = Net::HTTP::Get.new(uri.request_uri)
+        response = http.request(request)
+      end
+    end
+    
+  end
+  
+  class EventMachineLoop
+    def initialize loops
+      @total_loops, @loops = loops, 0
+    end
+    
+    def request!
+      http = EventMachine::HttpRequest.new('http://localhost:3000').get
+      http.callback { |http|
+        @loops += 1
+        http.response
+        all_requests_done? ? EM.stop : request!
+      }
+      http.errback { |http|
+        puts "EventMachine Request Error: #{http.error} #{http.inspect}"
+        EM.stop
+      }
+    end
+    
+    def all_requests_done?
+      @loops >= @total_loops
+    end
+  end
+  
+  benchmark.report "em sync" do
+    
+    looper = EventMachineLoop.new(n)
+    EventMachine.run {
+      looper.request!
+    }
+    
+  end
+  
+end

data/dev/test.rb

@@ -0,0 +1,13 @@
+require 'eventmachine'
+require 'net/http'
+require 'em-http'
+
+
+
+describe "Nothing" do
+  it "should endlessly run the eventmachine" do
+    #EventMachine.run {
+      # Test
+    #}
+  end
+end

data/dev/tracking.rb

@@ -0,0 +1,49 @@
+require 'benchmark'
+require 'active_support/all'
+
+{ :test => :a }.to_json
+
+n = 50_000
+
+Benchmark.bm do |benchmark|
+  
+  benchmark.report "hash tracking" do
+    
+    @trackings = []
+    0.upto(n) do
+      @trackings << {
+        :elapsed    => 0.23465723,
+        :start      => 1273651762534712,
+        :stop       => 1273658172365915,
+        :duration   => 23,
+        :benchmark  => "Some value",
+        :url        => "http://localhost:3000",
+        :path       => "/",
+        :method     => "GET",
+        :status     => 200
+      }
+    end
+    
+  end
+  
+  benchmark.report "hash-to_json tracking" do
+    
+    @trackings = []
+    0.upto(n) do
+      @trackings << {
+        :elapsed    => 0.23465723,
+        :start      => 1273651762534712,
+        :stop       => 1273658172365915,
+        :duration   => 23,
+        :benchmark  => "Some value",
+        :url        => "http://localhost:3000",
+        :path       => "/",
+        :method     => "GET",
+        :status     => 200
+      }.to_json
+    end
+    
+  end
+  
+  
+end

data/doc/BigBench.html

@@ -47,6 +47,7 @@
     <nav id="file-list-section" class="section">
   <h3 class="section-header">Defined In</h3>
   <ul>
+    <li>lib/bigbench/benchmark/looper.rb
     <li>lib/bigbench/benchmark.rb
     <li>lib/bigbench/bot.rb
     <li>lib/bigbench/configuration.rb
@@ -126,6 +127,8 @@
   
     <li><a href="./BigBench/Benchmark/Benchmark.html">BigBench::Benchmark::Benchmark</a>
   
+    <li><a href="./BigBench/Benchmark/Looper.html">BigBench::Benchmark::Looper</a>
+  
     <li><a href="./BigBench/Bot.html">BigBench::Bot</a>
   
     <li><a href="./BigBench/Configuration.html">BigBench::Configuration</a>
@@ -154,14 +157,14 @@
   
     <li><a href="./BigBench/Tracker/Tracker.html">BigBench::Tracker::Tracker</a>
   
+    <li><a href="./EventMachineLoop.html">EventMachineLoop</a>
+  
     <li><a href="./Float.html">Float</a>
   
     <li><a href="./Helpers.html">Helpers</a>
   
     <li><a href="./Object.html">Object</a>
   
-    <li><a href="./TestWebServer.html">TestWebServer</a>
-  
   </ul>
 </nav>
 
@@ -223,18 +226,19 @@
    <span class="ruby-identifier">get</span> <span class="ruby-string">&quot;/&quot;</span>
    <span class="ruby-identifier">get</span> <span class="ruby-string">&quot;/blog&quot;</span>
    <span class="ruby-identifier">get</span> <span class="ruby-string">&quot;/imprint&quot;</span>
+   <span class="ruby-identifier">get</span> <span class="ruby-string">&quot;/admin&quot;</span>, :<span class="ruby-identifier">basic_auth</span> =<span class="ruby-operator">&gt;</span> [<span class="ruby-string">'admin'</span>, <span class="ruby-string">'secret'</span>]
 <span class="ruby-keyword">end</span>
 
 <span class="ruby-identifier">benchmark</span> <span class="ruby-string">&quot;login and logout&quot;</span> =<span class="ruby-operator">&gt;</span> <span class="ruby-string">&quot;http://localhost:3000&quot;</span> <span class="ruby-keyword">do</span>
-   <span class="ruby-identifier">post</span> <span class="ruby-string">&quot;/login&quot;</span>, { :<span class="ruby-identifier">name</span> =<span class="ruby-operator">&gt;</span> <span class="ruby-string">&quot;test@user.com&quot;</span>, :<span class="ruby-identifier">password</span> =<span class="ruby-operator">&gt;</span> <span class="ruby-string">&quot;secret&quot;</span> }
-   <span class="ruby-identifier">post</span> <span class="ruby-string">&quot;http://localhost:3000/logout&quot;</span>
+   <span class="ruby-identifier">post</span> <span class="ruby-string">&quot;/login&quot;</span>,  :<span class="ruby-identifier">params</span> =<span class="ruby-operator">&gt;</span> { :<span class="ruby-identifier">name</span> =<span class="ruby-operator">&gt;</span> <span class="ruby-string">&quot;test@user.com&quot;</span>, :<span class="ruby-identifier">password</span> =<span class="ruby-operator">&gt;</span> <span class="ruby-string">&quot;secret&quot;</span> }
+   <span class="ruby-identifier">post</span> <span class="ruby-string">&quot;/logout&quot;</span>, :<span class="ruby-identifier">params</span> =<span class="ruby-operator">&gt;</span> { :<span class="ruby-identifier">name</span> =<span class="ruby-operator">&gt;</span> <span class="ruby-string">&quot;test@user.com&quot;</span> }
 <span class="ruby-keyword">end</span>
 </pre>
           
 
           
           <div class="method-source-code" id="benchmark-source">
-            <pre><span class="ruby-comment"># File lib/bigbench/benchmark.rb, line 111</span>
+            <pre><span class="ruby-comment"># File lib/bigbench/benchmark.rb, line 101</span>
 <span class="ruby-keyword">def</span> <span class="ruby-keyword">self</span>.<span class="ruby-identifier">benchmark</span>(<span class="ruby-identifier">options</span>)
   <span class="ruby-keyword">return</span> <span class="ruby-keyword">unless</span> <span class="ruby-identifier">block_given?</span>
   <span class="ruby-constant">Benchmark</span>.<span class="ruby-identifier">add</span>(<span class="ruby-identifier">options</span>, &amp;<span class="ruby-constant">Proc</span>.<span class="ruby-identifier">new</span>)
@@ -265,7 +269,7 @@
 
           
           <div class="method-source-code" id="benchmarks-source">
-            <pre><span class="ruby-comment"># File lib/bigbench/benchmark.rb, line 117</span>
+            <pre><span class="ruby-comment"># File lib/bigbench/benchmark.rb, line 107</span>
 <span class="ruby-keyword">def</span> <span class="ruby-keyword">self</span>.<span class="ruby-identifier">benchmarks</span>
   <span class="ruby-constant">Benchmark</span>.<span class="ruby-identifier">all</span>
 <span class="ruby-keyword">end</span></pre>
@@ -296,7 +300,7 @@
 
           
           <div class="method-source-code" id="check_config-21-source">
-            <pre><span class="ruby-comment"># File lib/bigbench/configuration.rb, line 104</span>
+            <pre><span class="ruby-comment"># File lib/bigbench/configuration.rb, line 111</span>
 <span class="ruby-keyword">def</span> <span class="ruby-keyword">self</span>.<span class="ruby-identifier">check_config!</span>
   <span class="ruby-identifier">raise</span> <span class="ruby-constant">Configuration</span><span class="ruby-operator">::</span><span class="ruby-constant">InvalidOptions</span>.<span class="ruby-identifier">new</span> <span class="ruby-keyword">unless</span> <span class="ruby-identifier">config</span>.<span class="ruby-identifier">valid?</span>
   <span class="ruby-keyword">true</span>
@@ -331,7 +335,7 @@
 
           
           <div class="method-source-code" id="config-source">
-            <pre><span class="ruby-comment"># File lib/bigbench/configuration.rb, line 99</span>
+            <pre><span class="ruby-comment"># File lib/bigbench/configuration.rb, line 106</span>
 <span class="ruby-keyword">def</span> <span class="ruby-keyword">self</span>.<span class="ruby-identifier">config</span>
   <span class="ruby-constant">Configuration</span>.<span class="ruby-identifier">config</span>
 <span class="ruby-keyword">end</span></pre>
@@ -359,9 +363,10 @@
           <p>Configure the benchmark by supplying a hash of options like this:</p>
 
 <pre class="ruby"><span class="ruby-constant">BigBench</span>.<span class="ruby-identifier">configure</span> = {
-  :<span class="ruby-identifier">duration</span> =<span class="ruby-operator">&gt;</span> <span class="ruby-value">10</span>.<span class="ruby-identifier">seconds</span>,
-  :<span class="ruby-identifier">output</span>   =<span class="ruby-operator">&gt;</span> <span class="ruby-string">&quot;test.ljson&quot;</span>,
-  :<span class="ruby-identifier">threads</span>  =<span class="ruby-operator">&gt;</span> <span class="ruby-value">20</span>
+  :<span class="ruby-identifier">duration</span>    =<span class="ruby-operator">&gt;</span> <span class="ruby-value">10</span>.<span class="ruby-identifier">seconds</span>,
+  :<span class="ruby-identifier">output</span>      =<span class="ruby-operator">&gt;</span> <span class="ruby-string">&quot;test.ljson&quot;</span>,
+  :<span class="ruby-identifier">users</span>       =<span class="ruby-operator">&gt;</span> <span class="ruby-value">20</span>,
+  :<span class="ruby-identifier">basic_auth</span>  =<span class="ruby-operator">&gt;</span> [<span class="ruby-string">'username'</span>, <span class="ruby-string">'secret'</span>]
 }
 </pre>
 
@@ -371,7 +376,7 @@
 
           
           <div class="method-source-code" id="configure-3D-source">
-            <pre><span class="ruby-comment"># File lib/bigbench/configuration.rb, line 90</span>
+            <pre><span class="ruby-comment"># File lib/bigbench/configuration.rb, line 97</span>
 <span class="ruby-keyword">def</span> <span class="ruby-keyword">self</span>.<span class="ruby-identifier">configure=</span>(<span class="ruby-identifier">config</span>)
   <span class="ruby-constant">Configuration</span>.<span class="ruby-identifier">configure</span>=(<span class="ruby-identifier">config</span>)
 <span class="ruby-keyword">end</span></pre>
@@ -402,7 +407,7 @@ one</p>
 
           
           <div class="method-source-code" id="duration-source">
-            <pre><span class="ruby-comment"># File lib/bigbench/benchmark.rb, line 122</span>
+            <pre><span class="ruby-comment"># File lib/bigbench/benchmark.rb, line 112</span>
 <span class="ruby-keyword">def</span> <span class="ruby-keyword">self</span>.<span class="ruby-identifier">duration</span>
   <span class="ruby-constant">Benchmark</span>.<span class="ruby-identifier">max_duration</span>
 <span class="ruby-keyword">end</span></pre>
@@ -440,7 +445,7 @@ end'</span>
 
           
           <div class="method-source-code" id="load_test-21-source">
-            <pre><span class="ruby-comment"># File lib/bigbench/runner.rb, line 45</span>
+            <pre><span class="ruby-comment"># File lib/bigbench/runner.rb, line 46</span>
 <span class="ruby-keyword">def</span> <span class="ruby-keyword">self</span>.<span class="ruby-identifier">load_test!</span>(<span class="ruby-identifier">test</span>)
   <span class="ruby-constant">BigBench</span><span class="ruby-operator">::</span><span class="ruby-constant">Benchmark</span>.<span class="ruby-identifier">reset!</span>
   <span class="ruby-identifier">eval</span>(<span class="ruby-identifier">test</span>)
@@ -473,7 +478,7 @@ end'</span>
 
           
           <div class="method-source-code" id="run-21-source">
-            <pre><span class="ruby-comment"># File lib/bigbench/runner.rb, line 31</span>
+            <pre><span class="ruby-comment"># File lib/bigbench/runner.rb, line 32</span>
 <span class="ruby-keyword">def</span> <span class="ruby-keyword">self</span>.<span class="ruby-identifier">run!</span>
   <span class="ruby-constant">BigBench</span><span class="ruby-operator">::</span><span class="ruby-constant">Output</span>.<span class="ruby-identifier">running_benchmarks</span>
   <span class="ruby-constant">Runner</span>.<span class="ruby-identifier">run!</span>
@@ -515,7 +520,7 @@ end'</span>
     <span class="ruby-constant">BigBench</span>.<span class="ruby-identifier">benchmarks</span>.<span class="ruby-identifier">each</span> <span class="ruby-keyword">do</span> <span class="ruby-operator">|</span><span class="ruby-identifier">benchmark</span><span class="ruby-operator">|</span>
       <span class="ruby-identifier">benchmark</span>.<span class="ruby-identifier">tracker</span>.<span class="ruby-identifier">trackings</span>.<span class="ruby-identifier">each</span> <span class="ruby-keyword">do</span> <span class="ruby-operator">|</span><span class="ruby-identifier">tracking</span><span class="ruby-operator">|</span>
         <span class="ruby-constant">BigBench</span><span class="ruby-operator">::</span><span class="ruby-constant">Output</span>.<span class="ruby-identifier">wrote_trackings</span>(<span class="ruby-identifier">counter</span>) <span class="ruby-keyword">if</span> <span class="ruby-identifier">counter</span> <span class="ruby-operator">%</span> <span class="ruby-value">100</span> <span class="ruby-operator">==</span> <span class="ruby-value">0</span>
-        <span class="ruby-identifier">file</span>.<span class="ruby-identifier">write</span> <span class="ruby-identifier">tracking</span> <span class="ruby-operator">+</span> <span class="ruby-string">&quot;\n&quot;</span>
+        <span class="ruby-identifier">file</span>.<span class="ruby-identifier">write</span> <span class="ruby-identifier">tracking</span>.<span class="ruby-identifier">to_json</span> <span class="ruby-operator">+</span> <span class="ruby-string">&quot;\n&quot;</span>
         <span class="ruby-identifier">counter</span> <span class="ruby-operator">+=</span> <span class="ruby-value">1</span>
       <span class="ruby-keyword">end</span>
     <span class="ruby-keyword">end</span>
@@ -599,7 +604,7 @@ end'</span>
   <span class="ruby-constant">BigBench</span>.<span class="ruby-identifier">benchmarks</span>.<span class="ruby-identifier">each</span> <span class="ruby-keyword">do</span> <span class="ruby-operator">|</span><span class="ruby-identifier">benchmark</span><span class="ruby-operator">|</span>
     <span class="ruby-identifier">benchmark</span>.<span class="ruby-identifier">tracker</span>.<span class="ruby-identifier">trackings</span>.<span class="ruby-identifier">each</span> <span class="ruby-keyword">do</span> <span class="ruby-operator">|</span><span class="ruby-identifier">tracking</span><span class="ruby-operator">|</span>
       <span class="ruby-constant">BigBench</span><span class="ruby-operator">::</span><span class="ruby-constant">Output</span>.<span class="ruby-identifier">wrote_trackings</span>(<span class="ruby-identifier">counter</span>) <span class="ruby-keyword">if</span> <span class="ruby-identifier">counter</span> <span class="ruby-operator">%</span> <span class="ruby-value">100</span> <span class="ruby-operator">==</span> <span class="ruby-value">0</span>
-      <span class="ruby-constant">BigBench</span><span class="ruby-operator">::</span><span class="ruby-constant">Store</span>.<span class="ruby-identifier">add_tracking</span>(<span class="ruby-identifier">tracking</span>)
+      <span class="ruby-constant">BigBench</span><span class="ruby-operator">::</span><span class="ruby-constant">Store</span>.<span class="ruby-identifier">add_tracking</span>(<span class="ruby-identifier">tracking</span>.<span class="ruby-identifier">to_json</span>)
       <span class="ruby-identifier">counter</span> <span class="ruby-operator">+=</span> <span class="ruby-value">1</span>
     <span class="ruby-keyword">end</span>
   <span class="ruby-keyword">end</span>