stella 0.5.1 → 0.5.3

data/README.txt

@@ -1,13 +1,20 @@
-Stella Tools - Management and reporting  tools for performance testing 
+Stella - Your Performance Testing Friend
+
+Release: 0.5.3-preview (2008-12-23)
+
+This is a PREVIEW release. Don't trust and double verify!
 
-Release: 0.5.1 (2008-12-23)
 
 == Prerequisites
 
 * Linux, *BSD, Solaris
 * Ruby 1.8.x or 1.9.x
 * Ruby Libraries
-  * 
+  * fastthread
+	* mongrel
+	* rspec
+	* rdoc
+	
 * One of:
   * Apache Bench
   * Siege
@@ -17,58 +24,111 @@ Release: 0.5.1 (2008-12-23)
 == Installation
 
 Get it in one of the following ways:
-  * GitHub: http://github.com/solutious/stella
-  * RubyForge: http://stella.rubyforge.org/
-  * gem install stella 
+* RubyForge: http://stella.rubyforge.org/
+* gem install stella 
   
 Use ab, siege, and httperf like you normally would with the addition of stella at the beginning (examples are below).
 
+=== Debian (and derivatives)
+
+Debian and its derivative (Ubunutu) handling packing a bit differently (see: http://pkg-ruby-extras.alioth.debian.org/rubygems.html). There are a couple errors to watch out for during the installation. The solutions are below:
+
+"no such file to load -- mkmf (LoadError)"
+  
+	apt-get install ruby1.8-dev
+
+"ERROR: RDoc documentation generator not installed!"
+  
+	apt-get install rdoc
+
+
 == Examples
 
-    # Run Apache Bench with a warmup and rampup from 100 to 300 virtual users in increments of 25
+Run Apache Bench with a warmup and rampup from 100 to 300 virtual users in increments of 25
+
   stella --warmup=0.5 --rampup=25,300 ab -c 100 -n 10000 http://stellaaahhhh.com/search?term=trooper
     
-    # Run Siege, repeat the test 5 times. Automatically creates a summary averages and standard deviations. 
-  stella --datapath=./stella --testruns=5  siege -c 100 -r 100 -b http://stellaaahhhh.com/search?term=flock+of+seagulls
-  
-  
+
+Run Siege, repeat the test 5 times. Automatically creates a summary averages and standard deviations. 
+
+  stella --agent=ff-3-osx --testruns=5  siege -c 100 -r 100 -b http://stellaaahhhh.com/search?term=flock+of+seagulls
+    
+
+Run Httperf like you normally would (but all the test data will be collected for you)
+
+  stella httperf --hog --client=0/1 --server=127.0.0.1 --port=5600 --uri=/ --rate=50  --num-conns=3000 --timeout=5
+
+
 == Sample Output
 
-  $ stella -f csv -x 5 -w -m "httpd 2.2.9-prefork" siege -c 125 -r 100 -b http://stella:5600/
-	Writing test data to: ./stella/testruns/2008-12-23/test-006
+  $ stella -f csv -x 5 -w 0.75 -r 25,125 -m "httpd 2.2.9-prefork" siege -c 75 -r 10 -b http://stella:5600/
+	Writing test data to: stella/testruns/2008-12-23/test-054
+
+	  Warmup:       3750@37/1   100%    264.29/s    0.140s    0.024MB/s    0.340MB   14.000s .
 
-	    Warmup:    125@12      1.1700s 102.56/s   0.1100s (100.00%)  .
+	 -------------------------------------------------------------------
+	                 REQ@VU/s  AVAIL       REQ/s     RTIME       DATA/s       DATA      TIME
+
+	  Run 01:       7500@75/1   100%    345.30/s    0.210s    0.032MB/s    0.690MB   21.720s
+	  Run 02:       7500@75/1   100%    360.58/s    0.200s    0.033MB/s    0.690MB   20.800s
+	  Run 03:       7500@75/1   100%    359.02/s    0.210s    0.033MB/s    0.690MB   20.890s
+	 -------------------------------------------------------------------
+	   Total:      22500@73     100%    354.97/s    0.207s    0.033MB/s    2.070MB   63.410s
+	 Std Dev:                             6.86/s    0.005s    0.001MB/s               0.414s
+
+	  Run 04:      10000@100/1  100%    384.47/s    0.260s    0.035MB/s    0.920MB   26.010s
+	  Run 05:      10000@100/1  100%    385.06/s    0.260s    0.035MB/s    0.920MB   25.970s
+	  Run 06:      10000@100/1  100%    380.95/s    0.260s    0.035MB/s    0.920MB   26.250s
+	 -------------------------------------------------------------------
+	   Total:      30000@98     100%    383.49/s    0.260s    0.035MB/s    2.760MB   78.230s
+	 Std Dev:                             1.81/s    0.000s    0.000MB/s               0.124s
+
+	  Run 07:      12500@125/1  100%    397.20/s    0.310s    0.036MB/s    1.140MB   31.470s
+	  Run 08:      12500@125/1  100%    397.08/s    0.310s    0.036MB/s    1.140MB   31.480s
+	  Run 09:      12500@125/1  100%    397.58/s    0.310s    0.036MB/s    1.140MB   31.440s
+	 -------------------------------------------------------------------
+	   Total:      37500@123    100%    397.29/s    0.310s    0.036MB/s    3.420MB   94.390s
+	 Std Dev:                             0.21/s    0.000s    0.000MB/s               0.017s
 
-	    Run 01:   1250@125     3.4800s 359.20/s   0.3200s (100.00%)  
-	    Run 02:   1250@125     3.4400s 363.37/s   0.3200s (100.00%)  
-	    Run 03:   1250@125     5.0000s 250.00/s   0.4700s (100.00%)  
-	    Run 04:   1250@125     3.3300s 375.38/s   0.3100s (100.00%)  
-	    Run 05:   1250@125     3.4100s 366.57/s   0.3100s (100.00%)  
 	 -------------------------------------------------------------------
-	 Std Dev:                  0.6359s  46.76/s   0.0622s
-	   Total:     6250@116    18.6600s 342.90/s   0.3460s (100.00%)  
+	   Total:      90000@98     100%    378.58/s    0.259s    0.035MB/s    8.250MB  236.030s
+	 Std Dev:                            18.09/s    0.042s    0.002MB/s               4.225s
+
 
 All test data is collected under ./stella (this can be changed with the parameter --datapath):
+
   $ ls -l ./stella/testruns/2008-12-23/
-  test-001   test-002   test-003   test-004   test-005   test-006
+  test-001   test-002   test-003   test-004   test-005   test-006  ...  test-054
   
+
 A symbolic link points to the most recent test:
+
   $ ls -l ./stella/latest/
   ID.txt    MESSAGE.txt  SUMMARY.csv  run01    run02    run03    run04    run05    warmup
   
+
 Each run directory contains all associated data, including the command and configuration
+
   $ ls -l ./stella/latest/run01/
     COMMAND.txt    STDOUT.txt    siege.log    STDERR.txt    SUMMARY.csv    siegerc
     
   
+== Known Issues
+
+* The output for the REQ@VU/s columns is a work in progress. It's not aligned across tools and it will likely change in the next release. 
+* The summary data has not been audited. Don't trust and double verify!
+* httperf is functional but needs a lot more testing (most dev was done with ab and siege).
+* The Ruby API has not been finalized. It's functional but there's no example because it is subject to change. 
+* There are no specs. 
 
 == Report an issue
 
 Email issues and bugs to stella@solutious.com
 
-== More Information
 
-http://stella.github.com/
+== Even More Information
+
+http://www.youtube.com/watch?v=wmq-JDonTpc
 
 == License
 

data/Rakefile

@@ -31,7 +31,12 @@ require 'rake/rdoctask'
 require 'fileutils'
 include FileUtils
 
-version = "0.5.1"
+STELLA_HOME = File.expand_path(File.join(File.dirname(__FILE__)))
+$: << File.join(STELLA_HOME, 'lib')
+
+require 'stella'
+version = Stella::VERSION.to_s
+
 name = "stella"
 
 spec = Gem::Specification.new do |s|

data/lib/stella.rb

@@ -10,6 +10,7 @@ require 'utils/httputil'
 require 'utils/crypto-key'
 require 'utils/fileutil'
 require 'utils/mathutil'
+require 'utils/escape'
 
 # Common dependencies
 require 'stella/support'
@@ -21,8 +22,8 @@ require 'stella/text'
 require 'stella/logger'
 require 'stella/response'
 require 'stella/test/definition'
-require 'stella/test/runsummary'
-require 'stella/test/testsummary'
+require 'stella/test/run/summary'
+require 'stella/test/summary'
 
 # Commands
 require 'stella/command/base'
@@ -38,13 +39,13 @@ require 'stella/adapter/httperf'
 #  A friend in performance testing. 
 #
 module Stella 
-  LOGGER = Stella::Logger.new(:debug=>true)
+  LOGGER = Stella::Logger.new(:debug=>false)
   TEXT = Stella::Text.new('en')
   
   module VERSION #:nodoc:
     MAJOR = 0.freeze unless defined? MAJOR
     MINOR = 5.freeze unless defined? MINOR
-    TINY  = 1.freeze unless defined? TINY
+    TINY  = 3.freeze unless defined? TINY
     def self.to_s
       [MAJOR, MINOR, TINY].join('.')
     end

data/lib/stella/adapter/ab.rb

@@ -86,9 +86,9 @@ module Stella
           # because it applies the load factor. 
           value = self.send(canon)
           if (value.is_a? Array)
-            value.each { |el| command << "-#{cannon} '#{el}' " }
+            value.each { |el| command << "-#{canon} #{EscapeUtil.shell_single_word(el.to_s)} " }
           else
-            command << "-#{canon} '#{value}' "
+            command << "-#{canon} #{EscapeUtil.shell_single_word(value.to_s)} "
           end
 
         end
@@ -168,7 +168,19 @@ module Stella
 
 
       
-
+      def add_header(name, value)
+        @H ||= []
+        @H << "#{name}: #{value}"
+      end
+      
+      def user_agent=(list=[])
+        return unless list && !list.empty?
+        list = list.to_ary
+        list.each do |agent|
+          add_header("User-Agent", agent)
+        end
+      end
+      
       def vusers
         c || 0
       end
@@ -240,7 +252,22 @@ module Stella
           raw[n.to_sym] = v.to_f  unless raw.has_key? n.to_sym
         }
 
-        stats = Stella::Test::RunSummary.new
+        # Document Path:          /
+        # Document Length:        96 bytes
+        # 
+        # Concurrency Level:      75
+        # Time taken for tests:   2.001 seconds
+        # Complete requests:      750
+        # Failed requests:        0
+        # Write errors:           0
+        # Total transferred:      174000 bytes
+        # HTML transferred:       72000 bytes
+        # Requests per second:    374.74 [#/sec] (mean)
+        # Time per request:       200.138 [ms] (mean)
+        # Time per request:       2.669 [ms] (mean, across all concurrent requests)
+        # Transfer rate:          84.90 [Kbytes/sec] received
+
+        stats = Stella::Test::Run::Summary.new
 
         unless raw.empty? || !raw.has_key?(:time_taken_for_tests)
 

data/lib/stella/adapter/base.rb

@@ -60,13 +60,27 @@ module Stella::Adapter
     end
     
     def name
-      @command_name
+      @name
+    end
+    
+    def rate
+      1
+    end
+    def vuser_rate
+      "#{vusers}/#{rate}"
     end
     
     def command
       raise Stella::TEXT.msg(:error_class_must_override, 'command')
     end
     
+    def add_header
+      raise Stella::TEXT.msg(:error_class_must_override, 'add_header')
+    end
+    
+    def user_agent=
+      raise Stella::TEXT.msg(:error_class_must_override, 'user_agent=')
+    end
     
   end
 end

data/lib/stella/adapter/httperf.rb

@@ -55,9 +55,9 @@ module Stella
           # because it applies the load factor. 
           value = self.send(canon)
           if (value.is_a? Array)
-            value.each { |el| command << "--#{canon.tr('_', '-')} '#{el}' " }
+            value.each { |el| command << "--#{canon.tr('_', '-')} #{EscapeUtil.shell_single_word(el.to_s)} " }
           else
-            command << "--#{canon.tr('_', '-')} '#{value}' "
+            command << "--#{canon.tr('_', '-')} #{EscapeUtil.shell_single_word(value.to_s)} "
           end
 
         end
@@ -102,7 +102,7 @@ module Stella
               
         opts.on('--add-header=S', String) do |v| @add_header ||= []; @add_header << v; end
         opts.on('--burst-length=N', Integer) do |v| @burst_length = v end
-        opts.on('--client=N', Integer) do |v| @client = v end
+        opts.on('--client=S', String) do |v| @client = v end
         opts.on('-d N', '--debug=N', Integer) do |v| @debug ||= 0; @debug = v end
         opts.on('--failure-status=N', Integer) do |v| @failure_status = v end
         
@@ -161,6 +161,22 @@ module Stella
         @name && !instance_variables.empty?
       end
       
+      def add_header(name=false, value=false)
+        # This is a hack since we have an instance variable called add_header.
+        # I figure this is the best of two evils because I'd rather keep the 
+        # instance variable naming consistent. 
+        return @add_header if !name && !value 
+        @add_header ||= []
+        @add_header << "#{name}: #{value}"
+      end
+      
+      def user_agent=(list=[])
+        return unless list && !list.empty?
+        list = list.to_ary
+        list.each do |agent|
+          add_header("User-Agent", agent)
+        end
+      end
       def vusers
          @rate
       end
@@ -176,6 +192,21 @@ module Stella
       def vuser_requests
         0
       end
+      def wsess
+        @wsess.join(',')
+      end
+      
+      def wset
+        @wset.join(',')
+      end
+      
+      
+      def wsesslog
+        @wsesslog.join(',')
+      end
+      def wlog
+        @wlog.join(',')
+      end
       
       #def concurrent
       #  (@concurrent * @load_factor).to_i
@@ -235,7 +266,7 @@ module Stella
         return unless stats_file
         
         raw = stats_file.readlines.join
-        stats = Stella::Test::RunSummary.new
+        stats = Stella::Test::Run::Summary.new
         
         raw.scan(/Request rate: (\d+?\.\d+?) req.s .(\d+?\.\d+?) ms.req./) do |rate,time|
           stats.transaction_rate = rate.to_f

data/lib/stella/adapter/siege.rb

@@ -95,9 +95,9 @@ module Stella
           # because it applies the load factor. 
           value = self.send(canon)
           if (value.is_a? Array)
-            value.each { |el| command << "--#{cannon.tr('_', '-')} '#{el}' " }
+            value.each { |el| command << "--#{canon.tr('_', '-')} #{EscapeUtil.shell_single_word(el.to_s)} " }
           else
-            command << "--#{canon.tr('_', '-')} '#{value}' "
+            command << "--#{canon.tr('_', '-')} #{EscapeUtil.shell_single_word(value.to_s)} "
           end
 
         end
@@ -118,24 +118,24 @@ module Stella
       def process_options(arguments)
         options = OpenStruct.new
         opts = OptionParser.new 
-        opts.on('-V', '--version') do |v| options.version = v end
-        opts.on('-h', '--help') do |v| options.help = v end
-        opts.on('-C', '--config') do |v| options.config = v end
-        opts.on('-v', '--verbose') do |v| options.verbose = v end
-        opts.on('-g', '--get') do |v| options.get = v end
-        opts.on('-l', '--log') do |v| options.log = v end
-        opts.on('-m S', '--mark=S', String) do |v| options.mark = v end
-        opts.on('-d N', '--delay=N', Float) do |v| options.delay = v end
-        opts.on('-H S', '--header=S', String) do |v| options.header ||= []; options.header << v end
+        opts.on('-V', '--version') do |v| @version = v end
+        opts.on('-h', '--help') do |v| @help = v end
+        opts.on('-C', '--config') do |v| @config = v end
+        opts.on('-v', '--verbose') do |v| @verbose = v end
+        opts.on('-g', '--get') do |v| @get = v end
+        opts.on('-l', '--log') do |v| @log = v end
+        opts.on('-m S', '--mark=S', String) do |v| @mark = v end
+        opts.on('-d N', '--delay=N', Float) do |v| @delay = v end
+        opts.on('-H S', '--header=S', String) do |v| @header ||= []; @header << v end
           
-        opts.on('-r N', '--reps=N', Integer) do |v| options.reps = v.to_i end
-        opts.on('-c N', '--concurrent=N', Integer) do |v| options.concurrent = v.to_i end
-        opts.on('-R S', '--rc=S', String) do |v| options.rc = v end
-        opts.on('-f S', '--file=S', String) do |v| options.file = v end
-        opts.on('-t S', '--time=S', String) do |v| options.time = v end
-        opts.on('-b', '--benchmark') do |v| options.benchmark = true;  end
-        opts.on('-i', '--internet') do |v| options.internet = true; end
-        opts.on('-A S', '--user-agent=S', String) do |v| options.user_agent = v end
+        opts.on('-r N', '--reps=N', Integer) do |v| @reps = v.to_i end
+        opts.on('-c N', '--concurrent=N', Integer) do |v| @concurrent = v.to_i end
+        opts.on('-R S', '--rc=S', String) do |v| @rc = v end
+        opts.on('-f S', '--file=S', String) do |v| @file = v end
+        opts.on('-t S', '--time=S', String) do |v| @time = v end
+        opts.on('-b', '--benchmark') do |v| @benchmark = true;  end
+        opts.on('-i', '--internet') do |v| @internet = true; end
+        opts.on('-A S', '--user-agent=S', String) do |v| @user_agent ||= []; @user_agent << v end
 
         raise "You cannot select both --internet and --benchmark" if options.internet && options.benchmark
 
@@ -150,7 +150,19 @@ module Stella
         raise InvalidArgument.new(badarg)
       end
 
-
+      
+      def add_header(name, value)
+        @header ||= []
+        @header << "#{name}: #{value}"
+      end
+      def user_agent=(list=[])
+        return unless list && !list.empty?
+        list = list.to_ary
+        @user_agent ||= []
+        @user_agent << list
+        @user_agent.flatten
+      end
+      
       def vusers
         concurrent || 0 
       end
@@ -272,23 +284,33 @@ module Stella
           raw[n.to_sym] = v.to_f
         }
 
-        stats = Stella::Test::RunSummary.new
-
+        stats = Stella::Test::Run::Summary.new
+        
+        # Transactions:		         750 hits
+        # Availability:		      100.00 %
+        # Elapsed time:		        2.33 secs
+        # Data transferred:	        0.07 MB
+        # Response time:		        0.21 secs
+        # Transaction rate:	      321.89 trans/sec
+        # Throughput:		        0.03 MB/sec
+        # Concurrency:		       67.49
+        # Successful transactions:         750
+        # Failed transactions:	           0
+        # Longest transaction:	        0.33
+        # Shortest transaction:	        0.10
+        
         stats.vusers = raw[:concurrency]
         stats.data_transferred = raw[:data_transferred]
-
-
         stats.elapsed_time = raw[:elapsed_time]
         stats.response_time = raw[:response_time]
-
-        #stats.shortest_transaction = raw[:shortest_transaction]
-        #stats.longest_transaction = raw[:longest_transaction]
-
         stats.transactions = raw[:transactions].to_i
         stats.transaction_rate = raw[:transaction_rate]
         stats.failed = raw[:failed_transactions].to_i
         stats.successful = raw[:successful_transactions].to_i
-        #stats.raw = raw if @global_options.debug
+        
+        #stats.shortest_transaction = raw[:shortest_transaction]
+        #stats.longest_transaction = raw[:longest_transaction]
+        
         stats
       end