stella 0.3.2

data/README.txt

@@ -0,0 +1,135 @@
+Stella - Your Performance Testing Friend
+
+Release: 0.3.2-preview (2008-12-23)
+
+This is a PREVIEW release. Don't trust and double verify!
+
+
+== Prerequisites
+
+* Linux, *BSD, Solaris
+* Ruby 1.8.x or 1.9.x
+* Ruby Libraries
+  * fastthread
+	* mongrel
+	* rspec
+	* rdoc
+	
+* One of:
+  * Apache Bench
+  * Siege
+  * Httperf
+
+
+== Installation
+
+Get it in one of the following ways:
+* RubyForge: http://stella.rubyforge.org/
+* gem install stella 
+  
+Use ab, siege, and httperf like you normally would with the addition of stella at the beginning (examples are below).
+
+=== Debian (and derivatives)
+
+Debian and its derivative (Ubunutu) handling packing a bit differently (see: http://pkg-ruby-extras.alioth.debian.org/rubygems.html). There are a couple errors to watch out for during the installation. The solutions are below:
+
+"no such file to load -- mkmf (LoadError)"
+  
+	apt-get install ruby1.8-dev
+
+"ERROR: RDoc documentation generator not installed!"
+  
+	apt-get install rdoc
+
+
+== Examples
+
+Run Apache Bench with a warmup and rampup from 100 to 300 virtual users in increments of 25
+
+  stella --warmup=0.5 --rampup=25,300 ab -c 100 -n 10000 http://stellaaahhhh.com/search?term=trooper
+    
+
+Run Siege, repeat the test 5 times. Automatically creates a summary averages and standard deviations. 
+
+  stella --agent=ff-3-osx --testruns=5  siege -c 100 -r 100 -b http://stellaaahhhh.com/search?term=flock+of+seagulls
+    
+
+Run Httperf like you normally would (but all the test data will be collected for you)
+
+  stella httperf --hog --client=0/1 --server=127.0.0.1 --port=5600 --uri=/ --rate=50  --num-conns=3000 --timeout=5
+
+
+== Sample Output
+
+  $ stella -f csv -x 5 -w 0.75 -r 25,125 -m "httpd 2.2.9-prefork" siege -c 75 -r 10 -b http://stella:5600/
+	Writing test data to: stella/testruns/2008-12-23/test-054
+
+	  Warmup:       3750@37/1   100%    264.29/s    0.140s    0.024MB/s    0.340MB   14.000s .
+
+	 -------------------------------------------------------------------
+	                 REQ@VU/s  AVAIL       REQ/s     RTIME       DATA/s       DATA      TIME
+
+	  Run 01:       7500@75/1   100%    345.30/s    0.210s    0.032MB/s    0.690MB   21.720s
+	  Run 02:       7500@75/1   100%    360.58/s    0.200s    0.033MB/s    0.690MB   20.800s
+	  Run 03:       7500@75/1   100%    359.02/s    0.210s    0.033MB/s    0.690MB   20.890s
+	 -------------------------------------------------------------------
+	   Total:      22500@73     100%    354.97/s    0.207s    0.033MB/s    2.070MB   63.410s
+	 Std Dev:                             6.86/s    0.005s    0.001MB/s               0.414s
+
+	  Run 04:      10000@100/1  100%    384.47/s    0.260s    0.035MB/s    0.920MB   26.010s
+	  Run 05:      10000@100/1  100%    385.06/s    0.260s    0.035MB/s    0.920MB   25.970s
+	  Run 06:      10000@100/1  100%    380.95/s    0.260s    0.035MB/s    0.920MB   26.250s
+	 -------------------------------------------------------------------
+	   Total:      30000@98     100%    383.49/s    0.260s    0.035MB/s    2.760MB   78.230s
+	 Std Dev:                             1.81/s    0.000s    0.000MB/s               0.124s
+
+	  Run 07:      12500@125/1  100%    397.20/s    0.310s    0.036MB/s    1.140MB   31.470s
+	  Run 08:      12500@125/1  100%    397.08/s    0.310s    0.036MB/s    1.140MB   31.480s
+	  Run 09:      12500@125/1  100%    397.58/s    0.310s    0.036MB/s    1.140MB   31.440s
+	 -------------------------------------------------------------------
+	   Total:      37500@123    100%    397.29/s    0.310s    0.036MB/s    3.420MB   94.390s
+	 Std Dev:                             0.21/s    0.000s    0.000MB/s               0.017s
+
+	 -------------------------------------------------------------------
+	   Total:      90000@98     100%    378.58/s    0.259s    0.035MB/s    8.250MB  236.030s
+	 Std Dev:                            18.09/s    0.042s    0.002MB/s               4.225s
+
+
+All test data is collected under ./stella (this can be changed with the parameter --datapath):
+
+  $ ls -l ./stella/testruns/2008-12-23/
+  test-001   test-002   test-003   test-004   test-005   test-006  ...  test-054
+  
+
+A symbolic link points to the most recent test:
+
+  $ ls -l ./stella/latest/
+  ID.txt    MESSAGE.txt  SUMMARY.csv  run01    run02    run03    run04    run05    warmup
+  
+
+Each run directory contains all associated data, including the command and configuration
+
+  $ ls -l ./stella/latest/run01/
+    COMMAND.txt    STDOUT.txt    siege.log    STDERR.txt    SUMMARY.csv    siegerc
+    
+  
+== Known Issues
+
+* The output for the REQ@VU/s columns is a work in progress. It's not aligned across tools and it will likely change in the next release. 
+* The summary data has not been audited. Don't trust and double verify!
+* httperf is functional but needs a lot more testing (most dev was done with ab and siege).
+* The Ruby API has not been finalized. It's functional but there's no example because it is subject to change. 
+* There are no specs. 
+
+== Report an issue
+
+Email issues and bugs to stella@solutious.com
+
+
+== Even More Information
+
+http://www.youtube.com/watch?v=wmq-JDonTpc
+
+== License
+
+See LICENSE.txt

data/Rakefile

@@ -0,0 +1,100 @@
+require 'rake'
+require 'spec/rake/spectask'
+
+desc "Run all specs"
+Spec::Rake::SpecTask.new('spec') do |t|
+	t.spec_files = FileList['spec/*_spec.rb']
+end
+
+desc "Print specdocs"
+Spec::Rake::SpecTask.new(:doc) do |t|
+	t.spec_opts = ["--format", "specdoc", "--dry-run"]
+	t.spec_files = FileList['spec/*_spec.rb']
+end
+
+desc "Run all examples with RCov"
+Spec::Rake::SpecTask.new('rcov') do |t|
+	t.spec_files = FileList['spec/*_spec.rb']
+	t.rcov = true
+	t.rcov_opts = ['--exclude', 'examples']
+end
+
+task :default => :spec
+
+######################################################
+
+require 'rake'
+require 'rake/testtask'
+require 'rake/clean'
+require 'rake/gempackagetask'
+require 'rake/rdoctask'
+require 'fileutils'
+include FileUtils
+
+STELLA_HOME = File.expand_path(File.join(File.dirname(__FILE__)))
+$: << File.join(STELLA_HOME, 'lib')
+
+require 'stella'
+version = Stella::VERSION.to_s
+
+name = "stella"
+
+spec = Gem::Specification.new do |s|
+	s.name = name
+	s.version = version
+	s.summary = "A friend in performance testing."
+	s.description = "Run Apache Bench, Siege, or httperf tests in batches and aggregate results."
+	s.author = "Delano Mandelbaum"
+	s.email = "delano@solutious.com"
+	s.homepage = "http://stella.solutious.com/"
+	s.executables = [ "stella" ]
+	s.rubyforge_project = "stella"
+  s.extra_rdoc_files  = ['README.txt']
+
+	s.add_dependency 'mongrel'
+	s.add_dependency 'rspec'
+	#s.add_dependency 'session'
+
+	s.platform = Gem::Platform::RUBY
+	s.has_rdoc = true
+	
+	s.files = %w(Rakefile) + Dir.glob("{bin,doc,lib,spec,support,vendor}/**/**/*")
+	
+	s.require_path = "lib"
+	s.bindir = "bin"
+end
+
+Rake::GemPackageTask.new(spec) do |p|
+	p.need_tar = true if RUBY_PLATFORM !~ /mswin/
+end
+
+
+task :install => [ :rdoc, :package ] do
+	sh %{sudo gem install pkg/#{name}-#{version}.gem}
+end
+
+task :uninstall => [ :clean ] do
+	sh %{sudo gem uninstall #{name}}
+end
+
+Rake::TestTask.new do |t|
+	t.libs << "spec"
+	t.test_files = FileList['spec/*_spec.rb']
+	t.verbose = true
+end
+
+Rake::RDocTask.new do |t|
+	t.rdoc_dir = 'doc'
+	t.title    = "stella, a friend in performance testing"
+	t.options << '--line-numbers' << '--inline-source' << '-A cattr_accessor=object'
+	t.options << '--charset' << 'utf-8'
+	t.rdoc_files.include('LICENSE.txt')
+	t.rdoc_files.include('README.txt')
+	t.rdoc_files.include('lib/utils/*.rb')
+	t.rdoc_files.include('lib/stella.rb')
+	t.rdoc_files.include('lib/stella/*.rb')
+	t.rdoc_files.include('lib/stella/**/*.rb')
+end
+
+CLEAN.include [ 'pkg', '*.gem', '.config', 'doc' ]
+

data/bin/stella

@@ -0,0 +1,12 @@
+#!/usr/bin/env ruby 
+
+STELLA_HOME = File.expand_path(File.join(File.dirname(__FILE__), '..'))
+$: << File.join(STELLA_HOME, 'lib')
+
+require 'stella'
+require 'stella/cli'
+
+## Create and run the application
+app = Stella::CLI.new(ARGV, STDIN)
+app.run
+

data/lib/stella.rb

@@ -0,0 +1,58 @@
+
+$: << File.join(STELLA_HOME, 'vendor', 'useragent', 'lib')
+
+
+require 'date'
+require 'open3'
+
+# Common utilities
+require 'utils/httputil'
+require 'utils/crypto-key'
+require 'utils/fileutil'
+require 'utils/mathutil'
+require 'utils/escape'
+
+# Common dependencies
+require 'stella/support'
+require 'stella/storable'
+require 'user_agent'
+
+# Common objects
+require 'stella/text'
+require 'stella/logger'
+require 'stella/response'
+require 'stella/test/definition'
+require 'stella/test/run/summary'
+require 'stella/test/summary'
+
+# Commands
+require 'stella/command/base'
+require 'stella/command/localtest'
+
+# Adapters
+require 'stella/adapter/base'
+require 'stella/adapter/ab'
+require 'stella/adapter/siege'
+require 'stella/adapter/httperf'
+
+# = Stella
+#  A friend in performance testing. 
+#
+module Stella 
+  LOGGER = Stella::Logger.new(:debug=>false)
+  TEXT = Stella::Text.new('en')
+  
+  module VERSION #:nodoc:
+    MAJOR = 0.freeze unless defined? MAJOR
+    MINOR = 3.freeze unless defined? MINOR
+    TINY  = 2.freeze unless defined? TINY
+    def self.to_s
+      [MAJOR, MINOR, TINY].join('.')
+    end
+    def self.to_f
+      self.to_s.to_f
+    end
+  end
+  
+end
+

data/lib/stella/adapter/ab.rb

@@ -0,0 +1,303 @@
+
+
+
+module Stella
+  module Adapter
+
+    #Usage: ab [options] [http[s]://]hostname[:port]/path
+    #Options are:
+    #    -n requests     Number of requests to perform
+    #    -c concurrency  Number of multiple requests to make
+    #    -t timelimit    Seconds to max. wait for responses
+    #    -b windowsize   Size of TCP send/receive buffer, in bytes
+    #    -p postfile     File containing data to POST. Remember also to set -T
+    #    -T content-type Content-type header for POSTing, eg.
+    #                    'application/x-www-form-urlencoded'
+    #                    Default is 'text/plain'
+    #    -v verbosity    How much troubleshooting info to print
+    #    -w              Print out results in HTML tables
+    #    -i              Use HEAD instead of GET
+    #    -x attributes   String to insert as table attributes
+    #    -y attributes   String to insert as tr attributes
+    #    -z attributes   String to insert as td or th attributes
+    #    -C attribute    Add cookie, eg. 'Apache=1234. (repeatable)
+    #    -H attribute    Add Arbitrary header line, eg. 'Accept-Encoding: gzip'
+    #                    Inserted after all normal header lines. (repeatable)
+    #    -A attribute    Add Basic WWW Authentication, the attributes
+    #                    are a colon separated username and password.
+    #    -P attribute    Add Basic Proxy Authentication, the attributes
+    #                    are a colon separated username and password.
+    #    -X proxy:port   Proxyserver and port number to use
+    #    -V              Print version number and exit
+    #    -k              Use HTTP KeepAlive feature
+    #    -d              Do not show percentiles served table.
+    #    -S              Do not show confidence estimators and warnings.
+    #    -g filename     Output collected data to gnuplot format file.
+    #    -e filename     Output CSV file with percentages served
+    #    -r              Don't exit on socket receive errors.
+    #    -h              Display usage information (this message)
+    #    -Z ciphersuite  Specify SSL/TLS cipher suite (See openssl ciphers)
+    #    -f protocol     Specify SSL/TLS protocol (SSL2, SSL3, TLS1, or ALL)
+    class ApacheBench < Stella::Adapter::Base
+      
+      
+      attr_accessor :n, :c, :t, :b, :p, :T, :v, :w, :i, :x, :z, :y
+      attr_accessor :C, :H, :A, :P, :X, :V, :k, :d, :S, :e, :g, :r, :h, :Z, :f
+      
+      def initialize(options={}, arguments=[])
+        super(options, arguments)
+        @private_variables = ['private_variables', 'name', 'arguments', 'load_factor', 'working_directory']
+        @c = 1
+        @n = 1
+        @name = 'ab'
+        @load_factor = 1
+      end
+      
+
+      def version
+        vsn = 0
+        text = ""
+        Open3.popen3("#{@name} -V") do |stdin, stdout, stderr| 
+          text = stdout.readlines.join
+          text.scan(/Version (\d+?\.\d+)/) { |v| vsn = v[0] }
+        end
+        vsn
+      end
+      
+      def before
+        
+        @e = @working_directory + "/ab-percentiles.log" 
+        @e = File.expand_path(@e)
+
+        @g = @working_directory + "/ab-requests.log" 
+        @g = File.expand_path(@g)
+
+      end
+      def command
+        raise CommandNotReady.new(self.class.to_s) unless ready?
+
+        command = "#{@name} "
+
+        instance_variables.each do |name|
+          canon = name.tr('@', '')        # instance_variables returns '@name'
+          next if @private_variables.member?(canon)
+
+          # It's important that we take the value from the getter method
+          # because it applies the load factor. 
+          value = self.send(canon)
+          if (value.is_a? Array)
+            value.each { |el| command << "-#{canon} #{EscapeUtil.shell_single_word(el.to_s)} " }
+          else
+            command << "-#{canon} #{EscapeUtil.shell_single_word(value.to_s)} "
+          end
+
+        end
+
+        command << (@arguments.map { |uri| "'#{uri}'" }).join(' ') unless @arguments.empty?
+        command
+      end
+      # loadtest
+      #
+      # True or false: is the call to ab a load test? If it's a call to help or version or
+      # to display the config this with return false. It's no reason for someone to make this 
+      # call through Stella but it's here for goodness sake. 
+      def loadtest?
+        !@arguments.empty?  # The argument is a URI
+      end
+      def ready?
+        (!self.loadtest?) || (@name && !instance_variables.empty? && !@arguments.empty?)
+      end
+      
+      
+      def process_options(arguments)
+        options = OpenStruct.new
+        opts = OptionParser.new 
+        opts.on('-v') do |v| options.v = true end
+        opts.on('-w') do |v| options.w = true end # TODO: Print a note that we don't parse the HTML results
+        opts.on('-i') do |v| options.i = true end
+        opts.on('-V') do |v| options.V = true end
+        opts.on('-k') do |v| options.k = true end
+        opts.on('-d') do |v| options.d = true end
+        opts.on('-S') do |v| options.S = true end
+        opts.on('-r') do |v| options.r = true end
+        opts.on('-h') do |v| options.h = true end
+        opts.on('-e S', String) do |v| options.e = v end
+        opts.on('-g S', String) do |v| options.g = v end
+        opts.on('-p S', String) do |v| options.p = v end
+        opts.on('-T S', String) do |v| options.t = v end
+        opts.on('-x S', String) do |v| options.x = v end
+        opts.on('-y S', String) do |v| options.y = v end
+        opts.on('-z S', String) do |v| options.z = v end
+        opts.on('-P S', String) do |v| options.P = v end
+        opts.on('-Z S', String) do |v| options.Z = v end
+        opts.on('-f S', String) do |v| options.f = v end
+        opts.on('-c N', Integer) do |v| options.c = v end
+        opts.on('-n N', Integer) do |v| options.n = v end
+        opts.on('-t N', Integer) do |v| options.t = v end
+        opts.on('-b N', Integer) do |v| options.b = v end
+        opts.on('-H S', String) do |v| options.H ||= []; options.H << v; end
+        opts.on('-C S', String) do |v| options.C ||= []; options.C << v; end
+        
+        # NOTE: parse! removes the options it finds in @arguments. It will leave
+        # all unnamed arguments and throw a fit about unknown ones. 
+        opts.parse!(arguments)
+        
+        options
+      rescue OptionParser::InvalidOption => ex
+        # We want to replace this text so we grab just the name of the argument
+        badarg = ex.message.gsub('invalid option: ', '')
+        raise InvalidArgument.new(badarg)
+      end
+
+      def after
+        # We want to maintain copies of all test output, even when the user has 
+        # supplied other path names so we'll copy the files from the testrun directory
+        # to the location specified by the user
+        [[@options.e, 'csv'], [@options.g, 'tsv']].each do |tuple|
+          if File.expand_path(File.dirname(tuple[0])) != File.expand_path(@runpath)
+            from = tuple[0]
+            to = @runpath + "/ab-#{tuple[1]}.log"
+            next unless File.exists?(from)
+            File.copy(from, to)
+          end
+        end
+
+        save_stats
+      end
+
+
+
+      
+      def add_header(name, value)
+        @H ||= []
+        @H << "#{name}: #{value}"
+      end
+      
+      def user_agent=(list=[])
+        return unless list && !list.empty?
+        list = list.to_ary
+        list.each do |agent|
+          add_header("User-Agent", agent)
+        end
+      end
+      
+      def vusers
+        c || 0
+      end
+      def vusers=(v)
+        ratio = vuser_requests
+        @c = v
+        @n = ratio * @c
+      end
+      def requests
+        n || 0
+      end
+      def requests=(v)
+        @n = v
+      end
+      def vuser_requests
+        ratio = 1
+        # The request ratio tells us how many requests will be
+        # generated per vuser. It helps us later when we need to
+        # warmp up and ramp up.
+        if @n > 0 && @c > 0
+          ratio = (@n.to_f / @c.to_f).to_f
+        # If concurrency isn't set, we'll assume the total number of requests
+        # is intended to be per request
+        elsif @n > 0
+         ratio = @n
+        end
+        ratio
+      end
+      def c
+        (@c * @load_factor).to_i
+      end
+      def n
+        (@n * @load_factor).to_i
+      end
+
+      def hosts
+        hosts = @arguments || []
+        #hosts << get_hosts_from_file
+        hosts = hosts.map{ |h| tmp = URI.parse(h.strip); "#{tmp.host}:#{tmp.port}" }
+        hosts
+      end
+
+      def paths
+        paths = @arguments || []
+        #hosts << get_hosts_from_file
+        paths = paths.map{ |h| tmp = URI.parse(h.strip); "#{tmp.path}?#{tmp.query}" }
+        paths
+      end
+
+
+      
+      # Apache bench writes the summary to STDOUT
+      def stats_file
+        File.new(stdout_path)
+      end
+
+      def stats
+        return unless stats_file
+        raw = {}
+        stats_file.each_line { |l|
+          l.chomp!
+          nvpair = l.split(':')
+          next unless nvpair && nvpair.size == 2
+          n = nvpair[0].strip.tr(' ', '_').downcase[/\w+/]
+          v = nvpair[1].strip[/[\.\d]+/]
+          
+          # Apache Bench outputs two fields with the name "Time per request".
+          # We want only the first one so we don't overwrite values.
+          raw[n.to_sym] = v.to_f  unless raw.has_key? n.to_sym
+        }
+
+        # Document Path:          /
+        # Document Length:        96 bytes
+        # 
+        # Concurrency Level:      75
+        # Time taken for tests:   2.001 seconds
+        # Complete requests:      750
+        # Failed requests:        0
+        # Write errors:           0
+        # Total transferred:      174000 bytes
+        # HTML transferred:       72000 bytes
+        # Requests per second:    374.74 [#/sec] (mean)
+        # Time per request:       200.138 [ms] (mean)
+        # Time per request:       2.669 [ms] (mean, across all concurrent requests)
+        # Transfer rate:          84.90 [Kbytes/sec] received
+
+        stats = Stella::Test::Run::Summary.new
+
+        unless raw.empty? || !raw.has_key?(:time_taken_for_tests)
+
+          stats.elapsed_time = raw[:time_taken_for_tests]
+
+          # We want this in MB, Apache Bench gives Bytes. 
+          stats.data_transferred = ((raw[:html_transferred] || 0) / 1_048_576)
+
+          # total_transferred is header data + response data (html_transfered)
+          stats.headers_transferred = ((raw[:total_transferred] || 0) / 1_048_576) - stats.data_transferred
+
+          # Apache Bench returns ms
+          stats.response_time = (raw[:time_per_request] || 0) / 1000
+          stats.transaction_rate = raw[:requests_per_second]
+
+          stats.vusers = raw[:concurrency_level].to_i
+          stats.successful = raw[:complete_requests].to_i
+          stats.failed = raw[:failed_requests].to_i
+
+          stats.transactions = stats.successful + stats.failed
+
+          #stats.raw = raw  if @global_options.debug
+        end
+
+        stats
+      end
+
+
+
+    end
+    
+  end
+end