stella 0.7.6.007 → 0.8.0.000

data/CHANGES.txt

@@ -1,18 +1,39 @@
 STELLA, CHANGES
 
-#### 0.8.0 (2009-12-??) ###############################
+#### 0.8.X (XXXX-XX-XX) ###############################
+
+* TODO: Force response block content type. 
+* TODO: Add assert methods to response blocks (reconsider)
+* TODO: Add option to dump full request/response output to a file by client
+* TODO: review cooking handling. Not always sent automatically. 
+* TODO: review :variables in URI elements
+* TODO: Stella::Testrun (no local analog for runid so it's set by the service)
+* TODO: global responses
+* TODO: request block conditions. 
+
+
+#### 0.8.0 (2010-01-16) ###############################
+
+NOTE: The configuration language changed slightly in this release. 
+See the README for more information on what needs to be updated. 
 
 * FIXED: HTTP auth failures
 * FIXED: Request timeouts should count towards failures
 * FIXED: Stella::Client ID was changing between requests in certain conditions
+* FIXED: HTTP delete now works correctly
+* FIXED: Missing rsequential method in Container
+* FIXED: Don't override user agent header when specified
+* FIXED: variables testplan example. 
+* FIXED: Print reports again when load test is interrupted 
+* FIXED: Sporadic ERB template error under high concurrency
+* CHANGE: A proper message displayed when no host specified
+* CHANGE: Don't append httpclient info to User-Agent header
+* CHANGE: param and header helper methods now return strings instead of Procs.
+* CHANGE: Verbosity level 2 now prints only params, 3 w/ headers, 4 w/ body
 * ADDED: follow responses can contain definition blocks
 * ADDED: Support for specifying default path prefix on CLI
 * ADDED: Support for specifying timeouts
-
-* TODO: Replace runtime procs with string templates
-* TODO: review cooking handling. Not always sent automatically. 
-* TODO: global responses
-* TODO: review :variables in URI elements
+* ADDED: Friendly error for empty request configs
 
 
 #### 0.7.7 (2009-12-05) ###############################

data/README.md

@@ -0,0 +1,132 @@
+# Stella - 0.8 BETA
+
+**Blame Stella for breaking your web application!**
+
+Stella is an integration and load testing tool. It fits well into an agile development process because the configuration syntax is simple yet powerful and a single config can be used for both kinds of tests (integration and load). Stella runs at the protocol-level which means it generates HTTP requests and parses the responses but does not perform any browser simulation (see Caveats).
+
+### Important Information Regarding Your Testplans
+
+*The testplan configuration syntax changed a little in 0.8. Most configs will not be affected. See "News" below for more info.*
+
+## Features
+
+* Support for simulating multiple usecases at the same time
+* Sophisticated response handling 
+* Automatic parsing of HTML, XML, XHTML, YAML, and JSON response bodies
+* Dynamic variable replacement 
+* Support for testing multiple sites simultaneously
+
+
+## Caveats
+
+There are a few known limitations:
+
+* *SHOW-STOPPER*: An upper limit of around 200-300 concurrent virtual HTTP clients. There is a threading issue in the HTTPClient library which appears under high load. 
+* *SHOW-STOPPER (for some)*: No support for browser or UI based tests (a la Watir or Selenium). If this is a show stopper for you, check out [WatirGrid](http://github.com/90kts/watirgrid)
+* *ANNOYING*: File uploads do not work with some HTTP servers (WEBrick)
+* *ANNOYING*: Lack of documentation (see examples/ directory)
+* *ANNOYING*: Reporting is limited to log files and command-line output. You need to make your own graphs. 
+
+
+## Examples
+
+
+### Testplan Configuration
+
+Every load testing tool worth its salt allows you to control the logical flow with some form of programming language. One of the advantages that open source tools have over commercial ones is that the languages used are generally well known whereas most commercial tools use proprietary languages. This makes it possible to define sophisticated and realistic tests.
+
+Stella test plans are defined in subset of the Ruby programming language. They also typically contain more than one usecase which is important when simulating realistic load. 
+
+    usecase "An Example Usecase" do
+      get '/some/path' do
+        param :what  => 'food'
+        param :where => 'iowa'
+        response 200 do
+          # code executed when the server returns a 200 response. 
+        end
+      end
+      
+      get '/a/dynamic/:path' do    
+        param :path => random(4)   # => http://host/a/dynamic/jrr1
+      end
+    end
+    
+    
+See the [examples/](http://github.com/solutious/stella/tree/0.8/examples/) directory and [Getting Started](http://solutious.com/projects/stella/getting-started/) for more information. 
+
+
+### Running Tests
+
+Stella is a command-line tool with two main commands: "verify" for integration tests and "generate" for load tests. 
+
+    # Verify a test plan is defined correctly
+    # by running a single user functional test.
+    $ stella verify -p examples/essentials/plan.rb http://stellaaahhhh.com/
+    
+    # Generate load using the same test plan. 
+    $ stella generate -p examples/essentials/plan.rb -c 50 -d 10m http://stellaaahhhh.com/
+    
+
+See <tt>$ stella -h</tt> and <tt>$ stella example</tt> for more info. 
+
+## News
+
+### 2010-01-16: sequential, random, rsequential etc... methods now return ERB-style templates instead of Procs
+
+Pre-0.8 would return a Proc that would be evaluated at request time:
+
+    get "/" do
+      param :salt => random(8)    # => "#<Proc0x13423c0 ...>"
+    end
+    
+In 0.8 and beyond, the same configuration will return a String:
+
+    get "/" do
+      param :salt => random(8)    # => "<%= random(8) %>"
+    end
+
+## Installation
+
+Get it in one of the following ways:
+     
+    $ gem install stella --source=http://gemcutter.org/
+    $ sudo gem install stella --source=http://gemcutter.org/
+    $ git clone git://github.com/solutious/stella.git
+
+You can also download via [tarball](http://github.com/solutious/stella/tarball/latest) or [zip](http://github.com/solutious/stella/zipball/latest). 
+
+
+## More Information
+
+* [Homepage](http://solutious.com/projects/stella)
+* [Codes](http://github.com/solutious/stella)
+* [RDocs](http://solutious.com/stella)
+* [Stellaaahhhh](http://stellaaahhhh.com)
+
+
+## Credits
+
+* [Delano Mandelbaum](http://solutious.com)
+
+
+## Thanks 
+
+* Harm Aarts for the great test case and feedback!
+* Kalin Harvey for keeping me on track.
+* Dave L, the best intern money can't buy. 
+* Peter McCurdy for the feedback and bug fixes. 
+
+
+## Related Projects
+
+* [WatirGrid](http://github.com/90kts/watirgrid)
+* [Watir](http://watir.com/)
+* [JMeter](http://jakarta.apache.org/jmeter/)
+* [Tsung](http://tsung.erlang-projects.org/)
+* [Grinder](http://grinder.sourceforge.net/)
+* [Pylot](http://www.pylot.org/)
+* [Trample](http://github.com/jamesgolick/trample)
+
+## License
+
+See LICENSE.txt

data/Rakefile

@@ -52,8 +52,8 @@ end
 #about 'Public release to rubyforge'
 task 'publish:gem' => [:package] do |t|
   sh <<-end
-    rubyforge add_release -o Any -a CHANGES.txt -f -n README.rdoc #{name} #{name} #{@spec.version} pkg/#{name}-#{@spec.version}.gem &&
-    rubyforge add_file -o Any -a CHANGES.txt -f -n README.rdoc #{name} #{name} #{@spec.version} pkg/#{name}-#{@spec.version}.tgz 
+    rubyforge add_release -o Any -a CHANGES.txt -f -n README.md #{name} #{name} #{@spec.version} pkg/#{name}-#{@spec.version}.gem &&
+    rubyforge add_file -o Any -a CHANGES.txt -f -n README.md #{name} #{name} #{@spec.version} pkg/#{name}-#{@spec.version}.tgz 
   end
 end
 
@@ -64,7 +64,7 @@ Rake::RDocTask.new do |t|
 	t.options << '--line-numbers' <<  '-A cattr_accessor=object'
 	t.options << '--charset' << 'utf-8'
 	t.rdoc_files.include('LICENSE.txt')
-	t.rdoc_files.include('README.rdoc')
+	t.rdoc_files.include('README.md')
 	t.rdoc_files.include('CHANGES.txt')
 	#t.rdoc_files.include('Rudyfile')  # why is the formatting f'd?
 	t.rdoc_files.include('bin/*')

data/Rudyfile

@@ -10,7 +10,7 @@ machines do
   end
   
   env :stage do
-                           
+                      
     role :app do             
       positions 2      
       user :root           
@@ -19,8 +19,8 @@ machines do
   
     role :gen do
       user :root
-      size 'm1.large'
-      ami 'ami-7133d018'
+      size 'm1.small'
+      ami 'ami-212ccf48'  # this is stale and the 64-bit one is broken
     end
     
     role :demo do
@@ -38,9 +38,11 @@ end
 commands do
   allow :apt_get, "apt-get", :y, :q
   allow :gem_install, "/usr/bin/gem", "install", :n, '/usr/bin', :y, :V, "--no-rdoc", "--no-ri"
+  allow :gem19_install, "/usr/local/bin/gem", "install", :n, '/usr/local/bin', :y, :V, "--no-rdoc", "--no-ri"
   allow :gem_sources, "/usr/bin/gem", "sources"
   allow :gem_uninstall, "/usr/bin/gem", "uninstall", :V
   allow :update_rubygems
+  allow :update_rubygems19
   allow :rake
   allow :thin
   allow :stella
@@ -123,13 +125,6 @@ routines do
     end
   end
   
-  install_github do
-    remote :root do
-      gem_sources :a, "http://gems.github.com"
-      gem_install 'solutious-stella'
-    end
-  end
-
   package_gem do
     local do
       rm :r, :f, 'pkg'
@@ -146,9 +141,15 @@ routines do
   
   install_gem do
     before :package_gem
+    local do
+      cd 'pkg'
+      gems = unsafely {ls "*gem"}
+      $gemfile = gems.first
+    end
     remote :root do
-      file_upload "pkg/stella-#{Stella::VERSION}.gem", "/tmp/"
-      gem_install "/tmp/stella-#{Stella::VERSION}.gem"
+      file_upload "pkg/#{$gemfile}", "/tmp/"
+      gem_install "/tmp/#{$gemfile}"
+      gem19_install "/tmp/#{$gemfile}"
     end
 
   end
@@ -203,13 +204,13 @@ routines do
       apt_get "install", "ruby1.8-dev", "rdoc", "libzlib-ruby", "rubygems"
       apt_get "install", "libfcgi-dev", "libfcgi-ruby1.8"
       apt_get "install", "joe", "siege", "httperf"
-      gem_sources :a, "http://gems.github.com"
+      gem_sources :a, "http://gemcutter.org/"
       mkdir :p, "/var/lib/gems/1.8/bin" # Doesn't get created, but causes Rubygems to fail
       gem_install "builder", "session"
-      gem_install 'hoe-seattlerb'
-      gem_install 'rubygems-update', "-v=1.3.4"
+      gem_install 'rubygems-update'
+      gem19_install 'rubygems-update'
       update_rubygems
-      gem_install 'hoe'
+      update_rubygems19
     }
   }
                            

data/bin/stella

@@ -49,6 +49,7 @@ class Stella::CLI::Definition
   global :'no-param', "Do not include __stella query parameter header" do
     true
   end
+  global :R, :remote
   global :E, :engine, String, "Specify a load engine (experimental)"
   global :o, :output, String, "Write output to the given file" do |v|
     String.disable_color
@@ -144,7 +145,6 @@ class Stella::CLI::Definition
   
   # ----------------------------------  STELLA MISCELLANEOUS  --------
   # ------------------------------------------------------------------
-
   before do |obj|
     #puts $$
     @start = Time.now

data/examples/cookies/plan.rb

@@ -29,7 +29,7 @@ usecase "Temporary Cookies" do
   
   # Here the cookie will contain the search term
   get "/search", "Search" do
-    param :what  => "<%= random(['Big', 'Beads', 'Joe']) %>"
+    param :what  => random(['Big', 'Beads', 'Joe'])
     response do
       puts "COOKIE: " << headers['Set-Cookie'].first
     end

data/examples/csvdata/plan.rb

@@ -1,4 +1,4 @@
-# Stella Test Plan - Reading CSV Data (2009-10-08)
+# Stella Test Plan - Reading CSV Data (2010-01-16)
 #
 # TO BE DOCUMENTED.
 # 
@@ -22,6 +22,11 @@ usecase "Reading CSV Data" do
     param :what  => sequential(:search_terms, 0)
     param :where => sequential(:search_terms, 1)
   end
+  
+  get "/search", "Search (sequential #3)" do
+    param :what  => sequential(:search_terms, 0)
+    param :where => sequential(:search_terms, 1)
+  end
 end
 
 # d93df136283f3867f462266a98675ce0b2f51b08

data/examples/dynamic/plan.rb

@@ -44,7 +44,6 @@ usecase "Dynamic Data" do
   # Access each listing page in reverse order
   get "/listing/:lid.yaml", "Reverse Sequential" do
     param :lid => rsequential(:listing_ids)
-    
     response 200 do
       repeat 5
     end

data/examples/variables/plan.rb

@@ -34,7 +34,5 @@ usecase "Form Example" do
     end
   end
   
-  get 
-  
 end
 

data/lib/stella/cli.rb

@@ -19,6 +19,13 @@ class Stella::CLI < Drydock::Command
     opts = {}
     opts[:hosts] = @hosts
     opts[:nowait] = true if @option.nowait
+    
+    if @global.remote
+      require 'bone'
+      s = Stella::Service.new Bone['STELLA_SOURCE'], Bone['STELLA_TOKEN']
+      Stella::Engine.service = s
+    end
+    
     [:'no-templates', :'no-stats', :'no-header', :'no-param'].each do |opt|
       opts[opt] = @global.send(opt) unless @global.send(opt).nil?
     end

data/lib/stella/client/container.rb

@@ -86,11 +86,10 @@ class Stella::Client
       # ERB BUG?: Under heavy threading, some calls
       # produce the error:
       # wrong number of arguments(1 for 0)
-      template = ERB.new(t)
-      v = template.result(binding)  
+      v = t.result(binding)  
     rescue => ex
       Stella.ld ex.message, ex.backtrace
-      t
+      t.to_s
     end
     
     def doc
@@ -232,93 +231,146 @@ class Stella::Client
       
     end
     
+    def path(*args)
+      input = File.join *args
+      File.exists?(input) ? input : File.join(@base_path, input)
+    end
+    
+    def read_file(*args)
+      f = file *args
+      f.read
+    end
+    
     def random(*args)
-      if Symbol === args.first
-        input, index = *args
-      elsif Array === args.first || args.size == 1
-        input = args.first
+      input, idxcol = *std_arg_processor(*args)
+    
+      if @random_value[input.object_id]
+        value = @random_value[input.object_id]
       else
-        input = args
+        value = case input.class.to_s
+        when "Symbol"
+          resource(input)
+        when "Array"
+          input
+        when "Range"
+          input.to_a
+        when "Proc"
+          input.call
+        when "Fixnum"
+          Stella::Utils.strand( input )
+        when "NilClass"
+          Stella::Utils.strand( rand(100) )
+        end
+        raise Stella::Testplan::Usecase::UnknownResource, input if value.nil?
+        Stella.ld "RANDVALUES: #{input} #{value.class} #{value.inspect}"
+        value = value[ rand(value.size) ] if value.is_a?(Array)
+        Stella.ld "SELECTED: #{value.class} #{value} "
+        @random_value[input.object_id] = value
       end
-        
       
-        if @random_value[input.object_id]
-          value = @random_value[input.object_id]
-        else
-          value = case input.class.to_s
-          when "Symbol"
-            resource(input)
-          when "Array"
-            input
-          when "Range"
-            input.to_a
-          when "Proc"
-            input.call
-          when "Fixnum"
-            Stella::Utils.strand( input )
-          when "NilClass"
-            Stella::Utils.strand( rand(100) )
-          end
-          raise Stella::Testplan::Usecase::UnknownResource, input if value.nil?
-          Stella.ld "RANDVALUES: #{input} #{value.class} #{value.inspect}"
-          value = value[ rand(value.size) ] if value.is_a?(Array)
-          Stella.ld "SELECTED: #{value.class} #{value} "
-          @random_value[input.object_id] = value
+      # The resource may be an Array of Arrays (e.g. a CSV file)
+      if value.is_a?(Array) && !idxcol.nil?
+        value = value[ idxcol ] 
+        Stella.ld "SELECTED INDEX: #{idxcol} #{value.inspect} "
+      end
+      
+      value
+      
+    end
+    
+    
+    
+    # NOTE: This is global across all users
+    def sequential(*args)
+      input, idxcol = *std_arg_processor(*args)
+            
+      if @sequential_value[input.object_id]
+        value = @sequential_value[input.object_id]
+      else
+        value = case input.class.to_s
+        when "Symbol"
+          ret = resource(input)
+          ret
+        when "Array"
+          input
+        when "Range"
+          input.to_a
+        when "Proc"
+          input.call
         end
-        
-        # The resource may be an Array of Arrays (e.g. a CSV file)
-        if value.is_a?(Array) && !index.nil?
-          value = value[ index ] 
-          Stella.ld "SELECTED INDEX: #{index} #{value.inspect} "
+        digest = value.object_id
+        if value.is_a?(Array)
+          idxrow = Stella::Client::Container.sequential_offset(digest, value.size-1)
+          value = value[ idxrow ] 
+          Stella.ld "SELECTED(SEQ): #{value} #{idxrow} #{input} #{digest}"
+        end
+        # I think this needs to be updated for global_sequential:
+        @sequential_value[input.object_id] = value
+      end
+      # The resource may be an Array of Arrays (e.g. a CSV file)
+      if value.is_a?(Array) && !idxcol.nil?
+        value = value[ idxcol ] 
+        Stella.ld "SELECTED INDEX: #{idxcol} #{value.inspect} "
+      end
+      
+      value
+      
+    end
+    
+
+    # NOTE: This is global across all users
+    def rsequential(*args)
+      input, idxcol = *std_arg_processor(*args)
+      
+      if @rsequential_value[input.object_id]
+        value = @rsequential_value[input.object_id]
+      else
+        value = case input.class.to_s
+        when "Symbol"
+          ret = resource(input)
+          ret
+        when "Array"
+          input
+        when "Range"
+          input.to_a
+        when "Proc"
+          input.call
+        end
+        digest = value.object_id
+        if value.is_a?(Array)
+          idxrow = Stella::Client::Container.rsequential_offset(digest, value.size-1)
+          value = value[ idxrow ] 
+          Stella.ld "SELECTED(RSEQ): #{value} #{idxrow} #{input} #{digest}"
         end
         
-        value
+        # I think this needs to be updated for global_sequential:
+        @rsequential_value[input.object_id] = value
+      end
+      # The resource may be an Array of Arrays (e.g. a CSV file)
+      if value.is_a?(Array) && !idxcol.nil?
+        value = value[ idxcol ] 
+        Stella.ld "SELECTED INDEX: #{idxcol} #{value.inspect} "
+      end
+      value
       
     end
     
     
-    # NOTE: This is global across all users
-    def sequential(*args)
+    private 
+    
+    def std_arg_processor(*args)
+      input, idxcol = nil, nil
       if Symbol === args.first
-        input, index = *args
+        input, idxcol = *args
       elsif Array === args.first || args.size == 1
         input = args.first
       else
         input = args
       end
-      
-        if @sequential_value[input.object_id]
-          value = @sequential_value[input.object_id]
-        else
-          value = case input.class.to_s
-          when "Symbol"
-            ret = resource(input)
-            ret
-          when "Array"
-            input
-          when "Range"
-            input.to_a
-          when "Proc"
-            input.call
-          end
-          digest = value.object_id
-          if value.is_a?(Array)
-            index = Stella::Client::Container.sequential_offset(digest, value.size-1)
-            value = value[ index ] 
-          end
-          Stella.ld "SELECTED(SEQ): #{value} #{index} #{input} #{digest}"
-          # I think this needs to be updated for global_sequential:
-          @sequential_value[input.object_id] = value
-        end
-        # The resource may be an Array of Arrays (e.g. a CSV file)
-        if value.is_a?(Array) && !index.nil?
-          value = value[ index ] 
-          Stella.ld "SELECTED INDEX: #{index} #{value.inspect} "
-        end
-        value
-      
+      [input, idxcol]
     end
     
+    
   end
   
 end

data/lib/stella/client.rb

@@ -13,17 +13,21 @@ module Stella
     include Gibbler::Complex
     include Observable
     
-    attr_reader :client_id
+    attr_reader :index
     attr_accessor :base_uri
     attr_accessor :proxy
+    attr_accessor :created 
     
-    def initialize(base_uri=nil, client_id=1, opts={})
+    gibbler :opts, :index, :base_uri, :proxy, :nowait, :created
+    
+    def initialize(base_uri=nil, index=1, opts={})
+      @created = Time.now.to_f
       opts = {
         :'no-templates' => false
       }.merge! opts
       @opts = opts
-      @base_uri, @client_id = base_uri, client_id
-      #@cookie_file = File.new("cookies-#{client_id}", 'w')
+      @base_uri, @index = base_uri, index
+      #@cookie_file = File.new("cookies-#{index}", 'w')
       @proxy = OpenStruct.new
     end
     def execute(usecase, &stat_collector)
@@ -47,10 +51,10 @@ module Stella
           # This is for the values that were "set"
           # in the part before the response body.
           prepare_resources(container, req.resources)
-        
+          
           params = prepare_params(container, req.params)
           headers = prepare_headers(container, req.headers)
-        
+          
           container.params, container.headers = params, headers
           
           uri = build_request_uri req.uri, params, container
@@ -81,7 +85,7 @@ module Stella
           Benelux.add_thread_tags :retry => counter
           Benelux.add_thread_tags :stella_id => stella_id
           
-          container.unique_id = stella_id[0..10]
+          container.unique_id = stella_id
           
           params['__stella'] = container.unique_id unless @opts[:'no-param']
           headers['X-Stella-ID'] = container.unique_id unless @opts[:'no-header']
@@ -175,7 +179,12 @@ module Stella
   private
     # We use a method so we can time it with Benelux
     def send_request(http_client, usecase, meth, uri, req, params, headers, container, counter)
-      container.response = http_client.send(meth, uri, params, headers) # booya!
+      if meth == "delete"
+        args = [meth, uri, headers]
+      else
+        args = [meth, uri, params, headers]
+      end
+      container.response = http_client.send(*args) # booya!
     end
     
     def update(kind, *args)
@@ -204,7 +213,7 @@ module Stella
       }
       http_client = HTTPClient.new opts
       http_client.set_proxy_auth(@proxy.user, @proxy.pass) if @proxy.user
-      http_client.debug_dev = STDOUT if Stella.debug? 
+      http_client.debug_dev = STDOUT if Stella.debug? && Stella.stdout.lev >= 3
       http_client.protocol_version = "HTTP/1.1"
       http_client.ssl_config.verify_mode = ::OpenSSL::SSL::VERIFY_NONE
       http_client
@@ -221,9 +230,8 @@ module Stella
       newh = {}
       #Stella.ld "PREPARE HEADERS: #{headers}"
       hashobj.each_pair do |n,v|
-        v = container.instance_eval &v if v.is_a?(Proc)
         unless @opts[:'no-templates']
-          v = container.parse_template v if String === v
+          v = container.parse_template v
         end
         v = extra.call(v) unless extra.nil?
         newh[n] = v