bigbench 0.0.2 → 0.0.3

data/README.textile

@@ -42,6 +42,7 @@ benchmark "login and logout" => "http://localhost:3000" do
     post "/logout", :params => { :name => "test@user.com" }
 end
 
+post_process :statistics
 
 
 
@@ -67,11 +68,11 @@ bc.. bigbench run bots example.rb redis_url:port redis_password
 
 p. This will upload the test receipt to all bots and make them run it. Every bot reports its results back to the redis and the local machine then combines, and writes them to the output file. So you test with the same receipts and get the same results, no matter if your testing from the local host or with multiple bots.
 
-
+!http://southdesign.github.com/bigbench/images/structure.png(BigBench Request Structure)!
 
 h2. Output
 
-How does the recorded output look like? It's in the @*.ljson*@ format which is nothing else but a textfile with a complete JSON object on every line. It looks like this:
+How does the recorded output look like? It's in the @*.ljson@ format which is nothing else but a textfile with a complete JSON object on every line. It looks like this:
 
 bc.. {"elapsed":0.002233,"start":1333981203.542233,"stop":1333981203.54279,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
 {"elapsed":0.00331,"start":1333981203.5434968,"stop":1333981203.5438669,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
@@ -95,6 +96,82 @@ p. The advantage with this file format is, that it can be parsed and computed ve
 
 
 
+h2. Post Processors
+
+After the benchmark has finished you can create hooks and write plugins that do something with the collected data. To setup a hook simply use the @post_process@ method to add a block or run a predefined plugin:
+
+bc.. # Run BigBench::PostProcessor::Statistics
+post_process :statistics
+
+# Run a block that could do anything
+post_process do
+
+    total_trackings, total_errors = 0, 0                
+    each_tracking do |tracking|
+        total_trackings += 1
+        total_errors    += 1 unless tracking[:status] == 200
+    end
+
+    Twitter.post "Just run BigBench with #{total_trackings} trackings and #{total_errors} errors."
+
+end
+
+p. It's very easy to write a post processor. The basic structure is like this:
+
+bc.. module BigBench
+    module PostProcessor
+        module SamplePostProcessor
+            
+            def self.run!
+                # Do whatever you want here
+            end
+            
+        end
+    end
+end
+
+p. You can hook it in with:
+
+bc.. post_process :sample_post_processor
+# or
+post_process BigBench::PostProcessor::SamplePostProcessor
+
+p. Contribute, create great post processors and send me a pull request!
+
+h3. Statistics
+
+p. The statistics post processor computes a simple overview of the benchmark and prints it to the terminal like this:
+
+bc.. BigBench Statistics
++---------------------------+------------------+---------+
+| Name                      | Value            | Percent |
++---------------------------+------------------+---------+
+| Total Requests:           | 52,469           | 100%    |
+| Total Errors:             | 0                | 0.0%    |
+|                           |                  |         |
+| Average Requests/Second:  | 437 Requests/sec |         |
+| Average Request Duration: | 1 ms             |         |
+|                           |                  |         |
+| Max Request Duration:     | 181 ms           |         |
+| Min Request Duration:     | 1 ms             |         |
+|                           |                  |         |
+| Status Codes:             |                  |         |
+|   200                     | 52469            | 100.0%  |
+|                           |                  |         |
+| HTTP Methods              |                  |         |
+|   get                     | 34980            | 66.7%   |
+|   post                    | 17489            | 33.3%   |
+|                           |                  |         |
+| URL Paths:                |                  |         |
+|   /                       | 34979            | 66.7%   |
+|   /basic/auth             | 17490            | 33.3%   |
++---------------------------+------------------+---------+
+19 rows in set
+
+
+
+
+
 h2. Load Comparison
 
 BigBench is awfully good at creating high loads on web servers. A quick benchmark comparison to "Apache's JMeter":http://jmeter.apache.org/ shows that BigBench is able to create 16% more load than JMeter.
@@ -117,3 +194,27 @@ h3. Test Results
 | Total Requests        | 48.014            | 55.484            |
 | Requests/sec          | 377               | 462               |
 | Percentages           | 100%%             | 116%              |
+
+
+h2. Version History
+
+h3. 0.3
+
+* Added post processors hook that run after the benchmark with a simple plugin structure
+* Added a first basic post processor that computes the benchmark statistics and prints them in the terminal
+* Added ability to execute a block of code after running the benchmark. The code can do anything usefully like send emails, post twitter notifications or startup new servers
+
+h3. 0.2
+
+* @Net::HTTP@ was too slow. Only reached 35% of Apache's JMeter load. Changed requesting structure to "eventmachine":https://github.com/eventmachine/eventmachine using "em-http-request":https://github.com/igrigorik/em-http-request
+* Compared to JMeter it can create 16% *more* load than JMeter now
+* Changed config option from @threads@ to @users@ due to a better understanding
+* Added basic auth support
+* Added params hashes for request content
+
+h3. 0.1
+
+* Initial Version using @Net::HTTP@
+* Local and bot testing
+* LJSON output
+* Global configuration

data/bigbench.gemspec

@@ -20,15 +20,18 @@ Gem::Specification.new do |s|
   s.rdoc_options << '--include' << 'lib/bigbench/help/'
 
   s.add_development_dependency "rspec"
-  s.add_development_dependency "active_support"
+  s.add_development_dependency "activesupport", '>= 3.2.0'
+  s.add_development_dependency "actionpack",    '>= 3.2.0'
   s.add_development_dependency "rack"
   s.add_development_dependency "thin"
   s.add_development_dependency "eventmachine"
   s.add_development_dependency "em-http-request"
   s.add_development_dependency "sinatra"
   
+  s.add_runtime_dependency "activesupport", '>= 3.2.0'
+  s.add_runtime_dependency "actionpack",    '>= 3.2.0'
   s.add_runtime_dependency "redis"
-  s.add_runtime_dependency "active_support"
   s.add_runtime_dependency "eventmachine"
   s.add_runtime_dependency "em-http-request"
+  s.add_runtime_dependency "hirb"
 end

data/doc/BigBench.html

@@ -54,6 +54,8 @@
     <li>lib/bigbench/executor.rb
     <li>lib/bigbench/fragment.rb
     <li>lib/bigbench/output.rb
+    <li>lib/bigbench/post_processor/statistics.rb
+    <li>lib/bigbench/post_processor.rb
     <li>lib/bigbench/runner.rb
     <li>lib/bigbench/store.rb
     <li>lib/bigbench/tracker.rb
@@ -88,8 +90,14 @@
     
     <li><a href="#method-c-load_test-21">::load_test!</a>
     
+    <li><a href="#method-c-post_process">::post_process</a>
+    
+    <li><a href="#method-c-post_processors">::post_processors</a>
+    
     <li><a href="#method-c-run-21">::run!</a>
     
+    <li><a href="#method-c-run_post_processors-21">::run_post_processors!</a>
+    
     <li><a href="#method-c-write_local_trackings_to_file-21">::write_local_trackings_to_file!</a>
     
     <li><a href="#method-c-write_store_trackings_to_file-21">::write_store_trackings_to_file!</a>
@@ -147,6 +155,18 @@
   
     <li><a href="./BigBench/Output.html">BigBench::Output</a>
   
+    <li><a href="./BigBench/PostProcessor.html">BigBench::PostProcessor</a>
+  
+    <li><a href="./BigBench/PostProcessor/Environment.html">BigBench::PostProcessor::Environment</a>
+  
+    <li><a href="./BigBench/PostProcessor/InvalidProcessor.html">BigBench::PostProcessor::InvalidProcessor</a>
+  
+    <li><a href="./BigBench/PostProcessor/Processor.html">BigBench::PostProcessor::Processor</a>
+  
+    <li><a href="./BigBench/PostProcessor/Statistics.html">BigBench::PostProcessor::Statistics</a>
+  
+    <li><a href="./BigBench/PostProcessor/Test.html">BigBench::PostProcessor::Test</a>
+  
     <li><a href="./BigBench/Runner.html">BigBench::Runner</a>
   
     <li><a href="./BigBench/Runner/NoBenchmarksDefined.html">BigBench::Runner::NoBenchmarksDefined</a>
@@ -240,8 +260,7 @@
           <div class="method-source-code" id="benchmark-source">
             <pre><span class="ruby-comment"># File lib/bigbench/benchmark.rb, line 101</span>
 <span class="ruby-keyword">def</span> <span class="ruby-keyword">self</span>.<span class="ruby-identifier">benchmark</span>(<span class="ruby-identifier">options</span>)
-  <span class="ruby-keyword">return</span> <span class="ruby-keyword">unless</span> <span class="ruby-identifier">block_given?</span>
-  <span class="ruby-constant">Benchmark</span>.<span class="ruby-identifier">add</span>(<span class="ruby-identifier">options</span>, &amp;<span class="ruby-constant">Proc</span>.<span class="ruby-identifier">new</span>)
+  <span class="ruby-constant">Benchmark</span>.<span class="ruby-identifier">add</span>(<span class="ruby-identifier">options</span>, &amp;<span class="ruby-constant">Proc</span>.<span class="ruby-identifier">new</span>) <span class="ruby-keyword">if</span> <span class="ruby-identifier">block_given?</span>
 <span class="ruby-keyword">end</span></pre>
           </div><!-- benchmark-source -->
           
@@ -269,7 +288,7 @@
 
           
           <div class="method-source-code" id="benchmarks-source">
-            <pre><span class="ruby-comment"># File lib/bigbench/benchmark.rb, line 107</span>
+            <pre><span class="ruby-comment"># File lib/bigbench/benchmark.rb, line 106</span>
 <span class="ruby-keyword">def</span> <span class="ruby-keyword">self</span>.<span class="ruby-identifier">benchmarks</span>
   <span class="ruby-constant">Benchmark</span>.<span class="ruby-identifier">all</span>
 <span class="ruby-keyword">end</span></pre>
@@ -407,7 +426,7 @@ one</p>
 
           
           <div class="method-source-code" id="duration-source">
-            <pre><span class="ruby-comment"># File lib/bigbench/benchmark.rb, line 112</span>
+            <pre><span class="ruby-comment"># File lib/bigbench/benchmark.rb, line 111</span>
 <span class="ruby-keyword">def</span> <span class="ruby-keyword">self</span>.<span class="ruby-identifier">duration</span>
   <span class="ruby-constant">Benchmark</span>.<span class="ruby-identifier">max_duration</span>
 <span class="ruby-keyword">end</span></pre>
@@ -462,6 +481,120 @@ end'</span>
       </div><!-- load_test-21-method -->
 
     
+      <div id="method-c-post_process" class="method-detail ">
+        
+        <div class="method-heading">
+          <span class="method-name">post_process</span><span
+            class="method-args">(processor = nil)</span>
+          <span class="method-click-advice">click to toggle source</span>
+        </div>
+        
+
+        <div class="method-description">
+          
+          <p>To setup a post processor simply do this:</p>
+
+<pre>post_process do
+  # Some code that is executed after the tests, like a database update, twitter post, email etc.
+end</pre>
+
+<p>Or use one of the predefined post processor or write one yourself:</p>
+
+<pre>post_process :statistics
+post_process BigBench::PostProcessor::Statistics
+post_process &quot;statistics&quot;</pre>
+
+<p>All the upper lines include the same post processor. Symbols and strings
+are camelized and constantized as the module name.</p>
+
+<h2 id="method-c-post_process-label-Available+Methods+in+Processors">Available Methods in Processors</h2>
+
+<p>Every post processor block or module supports the following methods and has
+full  <a
+href="http://api.rubyonrails.org/classes/ActionView/Helpers.html">ActionView::Helper</a>
+support.</p>
+<dl class="rdoc-list label-list"><dt>each_tracking
+<dd>
+<p>A method that iterates through every collected tracking. It automatically
+returns a hash with a single tracking of the following form:</p>
+
+<pre class="ruby">{
+  :<span class="ruby-identifier">elapsed</span>    =<span class="ruby-operator">&gt;</span> <span class="ruby-value">2.502132</span>,
+  :<span class="ruby-identifier">start</span>      =<span class="ruby-operator">&gt;</span> <span class="ruby-value">1333986292.1755981</span>,
+  :<span class="ruby-identifier">stop</span>       =<span class="ruby-operator">&gt;</span> <span class="ruby-value">1333986293.618884</span>,
+  :<span class="ruby-identifier">duration</span>   =<span class="ruby-operator">&gt;</span> <span class="ruby-value">1443</span>,
+  :<span class="ruby-identifier">benchmark</span>  =<span class="ruby-operator">&gt;</span> <span class="ruby-string">&quot;index page&quot;</span>,
+  :<span class="ruby-identifier">url</span>        =<span class="ruby-operator">&gt;</span> <span class="ruby-string">&quot;http://www.google.de/&quot;</span>,
+  :<span class="ruby-identifier">path</span>       =<span class="ruby-operator">&gt;</span> <span class="ruby-string">&quot;/&quot;</span>,
+  :<span class="ruby-identifier">method</span>     =<span class="ruby-operator">&gt;</span> <span class="ruby-string">&quot;get&quot;</span>,
+  :<span class="ruby-identifier">status</span>     =<span class="ruby-operator">&gt;</span> <span class="ruby-value">200</span>
+}
+</pre>
+
+<p>It can be used like this:</p>
+
+<pre class="ruby"><span class="ruby-identifier">post_process</span> <span class="ruby-keyword">do</span>
+
+  <span class="ruby-identifier">total_trackings</span>, <span class="ruby-identifier">total_errors</span> = <span class="ruby-value">0</span>, <span class="ruby-value">0</span>                
+  <span class="ruby-identifier">each_tracking</span> <span class="ruby-keyword">do</span> <span class="ruby-operator">|</span><span class="ruby-identifier">tracking</span><span class="ruby-operator">|</span>
+    <span class="ruby-identifier">total_trackings</span> <span class="ruby-operator">+=</span> <span class="ruby-value">1</span>
+    <span class="ruby-identifier">total_errors</span>    <span class="ruby-operator">+=</span> <span class="ruby-value">1</span> <span class="ruby-keyword">unless</span> <span class="ruby-identifier">tracking</span>[:<span class="ruby-identifier">status</span>] <span class="ruby-operator">==</span> <span class="ruby-value">200</span>
+  <span class="ruby-keyword">end</span>
+
+  <span class="ruby-constant">Twitter</span>.<span class="ruby-identifier">post</span> <span class="ruby-node">&quot;Just run BigBench with #{total_trackings} trackings and #{total_errors} errors.&quot;</span>
+
+<span class="ruby-keyword">end</span>
+</pre>
+</dd></dl>
+          
+
+          
+          <div class="method-source-code" id="post_process-source">
+            <pre><span class="ruby-comment"># File lib/bigbench/post_processor.rb, line 192</span>
+<span class="ruby-keyword">def</span> <span class="ruby-keyword">self</span>.<span class="ruby-identifier">post_process</span> <span class="ruby-identifier">processor</span> = <span class="ruby-keyword">nil</span>
+  <span class="ruby-identifier">raise</span> <span class="ruby-constant">PostProcessor</span><span class="ruby-operator">::</span><span class="ruby-constant">InvalidProcessor</span>.<span class="ruby-identifier">new</span> <span class="ruby-keyword">if</span> <span class="ruby-identifier">processor</span>.<span class="ruby-identifier">nil?</span> <span class="ruby-operator">&amp;&amp;</span> <span class="ruby-operator">!</span><span class="ruby-identifier">block_given?</span>
+  <span class="ruby-identifier">block_given?</span> <span class="ruby-operator">?</span> <span class="ruby-constant">PostProcessor</span>.<span class="ruby-identifier">add</span>(&amp;<span class="ruby-constant">Proc</span>.<span class="ruby-identifier">new</span>) <span class="ruby-operator">:</span> <span class="ruby-constant">PostProcessor</span>.<span class="ruby-identifier">add</span>(<span class="ruby-identifier">processor</span>)
+<span class="ruby-keyword">end</span></pre>
+          </div><!-- post_process-source -->
+          
+        </div>
+
+        
+
+        
+      </div><!-- post_process-method -->
+
+    
+      <div id="method-c-post_processors" class="method-detail ">
+        
+        <div class="method-heading">
+          <span class="method-name">post_processors</span><span
+            class="method-args">()</span>
+          <span class="method-click-advice">click to toggle source</span>
+        </div>
+        
+
+        <div class="method-description">
+          
+          <p>List all initialized post processors</p>
+          
+
+          
+          <div class="method-source-code" id="post_processors-source">
+            <pre><span class="ruby-comment"># File lib/bigbench/post_processor.rb, line 198</span>
+<span class="ruby-keyword">def</span> <span class="ruby-keyword">self</span>.<span class="ruby-identifier">post_processors</span>
+  <span class="ruby-constant">PostProcessor</span>.<span class="ruby-identifier">all</span>
+<span class="ruby-keyword">end</span></pre>
+          </div><!-- post_processors-source -->
+          
+        </div>
+
+        
+
+        
+      </div><!-- post_processors-method -->
+
+    
       <div id="method-c-run-21" class="method-detail ">
         
         <div class="method-heading">
@@ -494,6 +627,36 @@ end'</span>
       </div><!-- run-21-method -->
 
     
+      <div id="method-c-run_post_processors-21" class="method-detail ">
+        
+        <div class="method-heading">
+          <span class="method-name">run_post_processors!</span><span
+            class="method-args">()</span>
+          <span class="method-click-advice">click to toggle source</span>
+        </div>
+        
+
+        <div class="method-description">
+          
+          <p>Runs all initialized post processors after the trackings have been written</p>
+          
+
+          
+          <div class="method-source-code" id="run_post_processors-21-source">
+            <pre><span class="ruby-comment"># File lib/bigbench/post_processor.rb, line 203</span>
+<span class="ruby-keyword">def</span> <span class="ruby-keyword">self</span>.<span class="ruby-identifier">run_post_processors!</span>
+  <span class="ruby-constant">PostProcessor</span>.<span class="ruby-identifier">run!</span>
+<span class="ruby-keyword">end</span></pre>
+          </div><!-- run_post_processors-21-source -->
+          
+        </div>
+
+        
+
+        
+      </div><!-- run_post_processors-21-method -->
+
+    
       <div id="method-c-write_local_trackings_to_file-21" class="method-detail ">
         
         <div class="method-heading">

data/doc/BigBench/Benchmark.html

@@ -124,6 +124,18 @@
   
     <li><a href="../BigBench/Output.html">BigBench::Output</a>
   
+    <li><a href="../BigBench/PostProcessor.html">BigBench::PostProcessor</a>
+  
+    <li><a href="../BigBench/PostProcessor/Environment.html">BigBench::PostProcessor::Environment</a>
+  
+    <li><a href="../BigBench/PostProcessor/InvalidProcessor.html">BigBench::PostProcessor::InvalidProcessor</a>
+  
+    <li><a href="../BigBench/PostProcessor/Processor.html">BigBench::PostProcessor::Processor</a>
+  
+    <li><a href="../BigBench/PostProcessor/Statistics.html">BigBench::PostProcessor::Statistics</a>
+  
+    <li><a href="../BigBench/PostProcessor/Test.html">BigBench::PostProcessor::Test</a>
+  
     <li><a href="../BigBench/Runner.html">BigBench::Runner</a>
   
     <li><a href="../BigBench/Runner/NoBenchmarksDefined.html">BigBench::Runner::NoBenchmarksDefined</a>

data/doc/BigBench/Benchmark/Benchmark.html

@@ -127,6 +127,18 @@
   
     <li><a href="../../BigBench/Output.html">BigBench::Output</a>
   
+    <li><a href="../../BigBench/PostProcessor.html">BigBench::PostProcessor</a>
+  
+    <li><a href="../../BigBench/PostProcessor/Environment.html">BigBench::PostProcessor::Environment</a>
+  
+    <li><a href="../../BigBench/PostProcessor/InvalidProcessor.html">BigBench::PostProcessor::InvalidProcessor</a>
+  
+    <li><a href="../../BigBench/PostProcessor/Processor.html">BigBench::PostProcessor::Processor</a>
+  
+    <li><a href="../../BigBench/PostProcessor/Statistics.html">BigBench::PostProcessor::Statistics</a>
+  
+    <li><a href="../../BigBench/PostProcessor/Test.html">BigBench::PostProcessor::Test</a>
+  
     <li><a href="../../BigBench/Runner.html">BigBench::Runner</a>
   
     <li><a href="../../BigBench/Runner/NoBenchmarksDefined.html">BigBench::Runner::NoBenchmarksDefined</a>

data/doc/BigBench/Benchmark/Looper.html

@@ -127,6 +127,18 @@
   
     <li><a href="../../BigBench/Output.html">BigBench::Output</a>
   
+    <li><a href="../../BigBench/PostProcessor.html">BigBench::PostProcessor</a>
+  
+    <li><a href="../../BigBench/PostProcessor/Environment.html">BigBench::PostProcessor::Environment</a>
+  
+    <li><a href="../../BigBench/PostProcessor/InvalidProcessor.html">BigBench::PostProcessor::InvalidProcessor</a>
+  
+    <li><a href="../../BigBench/PostProcessor/Processor.html">BigBench::PostProcessor::Processor</a>
+  
+    <li><a href="../../BigBench/PostProcessor/Statistics.html">BigBench::PostProcessor::Statistics</a>
+  
+    <li><a href="../../BigBench/PostProcessor/Test.html">BigBench::PostProcessor::Test</a>
+  
     <li><a href="../../BigBench/Runner.html">BigBench::Runner</a>
   
     <li><a href="../../BigBench/Runner/NoBenchmarksDefined.html">BigBench::Runner::NoBenchmarksDefined</a>

data/doc/BigBench/Bot.html

@@ -117,6 +117,18 @@
   
     <li><a href="../BigBench/Output.html">BigBench::Output</a>
   
+    <li><a href="../BigBench/PostProcessor.html">BigBench::PostProcessor</a>
+  
+    <li><a href="../BigBench/PostProcessor/Environment.html">BigBench::PostProcessor::Environment</a>
+  
+    <li><a href="../BigBench/PostProcessor/InvalidProcessor.html">BigBench::PostProcessor::InvalidProcessor</a>
+  
+    <li><a href="../BigBench/PostProcessor/Processor.html">BigBench::PostProcessor::Processor</a>
+  
+    <li><a href="../BigBench/PostProcessor/Statistics.html">BigBench::PostProcessor::Statistics</a>
+  
+    <li><a href="../BigBench/PostProcessor/Test.html">BigBench::PostProcessor::Test</a>
+  
     <li><a href="../BigBench/Runner.html">BigBench::Runner</a>
   
     <li><a href="../BigBench/Runner/NoBenchmarksDefined.html">BigBench::Runner::NoBenchmarksDefined</a>