jtzemp-tourbus 0.0.9

data/MIT-LICENSE

@@ -0,0 +1,22 @@
+Copyright (c) 2008-2009 David Brady github@shinybit.com
+ 
+Permission is hereby granted, free of charge, to any person
+obtaining a copy of this software and associated documentation
+files (the "Software"), to deal in the Software without
+restriction, including without limitation the rights to use,
+copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the
+Software is furnished to do so, subject to the following
+conditions:
+ 
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+ 
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.

data/README.rdoc

@@ -0,0 +1,200 @@
+= TourBus
+
+Flexible and scalable website testing tool.
+
+== Authors
+
+* David Brady -- david.brady@leadmediapartners.com
+* Tim Harper -- tim.harper@leadmediapartners.com
+* James Britt -- james@neurogami.com
+* JT Zemp -- jtzemp@gmail.com
+
+
+== General Info
+
+TourBus is an intelligent website load testing tool. Allows for
+complicated testing scenarios including filling out forms, following
+redirects, handling cookies, and following links--all of the things
+you'd normally associate with a regression suite or integration
+testing tool. The difference is that TourBus also scales concurrently,
+and you can perform hundreds of complicated regression tests
+simultaneously in order to thoroughly load test your website.
+
+== Motivation
+
+I started writing TourBus because I needed flexibility and scalability
+in a website testing tool, and the extant tools all provided one but
+not the other. Selenium is ultraflexible but limited to the number of
+browsers you can have open at once, while Apache Bench is powerful and
+fast but limited to simple tests.
+
+TourBus lets you define complicated paths through your website, then
+execute those paths concurrently for stress testing.
+
+== Example
+
+To see TourBus in action, you need to write scripts. For lack of a
+better name, these are called Tours. 
+
+=== Example Tour 
+
+* Make a folder called tours and put a file in it called simple.rb. In
+  it write:
+
+      class Simple < Tour
+        def test_homepage
+          visit "http://#{@host}/"
+          assert_contain "My Home Page"
+        end
+      end
+
+* Files in ./tours should have classes that match their names. E.g.
+  "class BigHairyTest < Tour" belongs in ./tours/big_hairy_test.rb
+
+* Think Test::Unit. test_* methods will be found automagically.
+  setup() and teardown() methods will be executed at the appropriate
+  times.
+
+=== Example TourBus Run
+
+You want to invoke +tourbus+ from the parent directory of the @tours/@ folder.
+
+For example, if you have this project  tree ...
+
+    `-- contact_app
+        |-- README.rdoc
+        |-- contact_app.rb
+        `-- tours
+            |-- simple.rb
+            `-- tourbus.yml
+
+... then you execute  +tourbus+ from the +contact_app/+ directory.
+
+    tourbus -c 2 -n 3 simple
+
+That will run the +simple.rb+ tour file.
+
+It will create 2 concurrent Tour runners, each of which will run all
+of the methods in Simple three times.
+
+* You can specify multiple tours.
+
+  tourbus -c 2 -n 3 simple1 simple2 simple3
+
+* If you don't specify a tour, all tours in ./tours will be run.
+
+* tourbus --help will give you more information.
+
+* You can run tours and filter given tests.
+
+  tourbus -c 2 -n 3 simple -t test_login,test_logout
+
+Note that if you specify multiple tours and filter tests, the filtered
+tests will be run on all tours specified. If you do not specify a
+tour, the filtered tests will be run on all tours found in the
++./tours+ folder.
+
+=== Example TourWatch Run
+
+On the webserver, you can type 
+
+    tourwatch -c 4
+
+To begin running tourwatch. It's basically a stripped-down version of
+top with cheesy text graphs. (TourWatch's development cycles were
+included in the 2 days for TourBus.)
+
+* The -c option is for the total number of cores on the server. The
+  top app will cheerfully report a process as taking 392% CPU if it is
+  using 98% of four cores. This option is only necessary for making
+  the little text graphs scale correctly.
+
+* You can choose which processes to watch by passing a csv to -p:
+  
+    tourwatch -p ruby,mongrel
+
+  Each process name is a partial regexp, so the above would match
+  mongrel AND mongrel_rails, etc.
+
+* tourwatch --help will give you more information.
+
+== History and Status
+
+TourBus began life as a 2-day throwaway app. It is definitely an app
+whose development provides many opportunities for open-source
+contributors to make improvements. It is chock-full of brutal hacks,
+duplications, oversights, and kludges.
+
+== Hacks, Kludges, Known Issues, and Piles of Steaming Poo
+
+* If you give a tour a name that is pluralized, it won't work. This is
+  probably a bug worth fixing. The reason for it is that we take file
+  names and "classify" them, and e.g. "ranking_reports" becomes
+  "RankingReport", not "RankingReports". This is an artifact of
+  borrowing from Rails' activesupport libs and should probably be
+  fixed.
+
+* Mechanize 0.8 doesn't always play well together with TourBus. If you
+  get "connection refused" socket errors, try upgrading to Mechanize
+  0.9.
+
+* JRuby doesn't play well with Nokogiri. I have set the html_parser to
+  use hpricot, which should work around the issue for now.
+
+* There are no specs. Yikes! This is to my eternal shame because I'm
+  sort of a testing freak. Because TourBus *WAS* a testing tool, I
+  didn't put tests on it. I haven't put tests on it yet because I'm
+  not sure how to go about it. Instead of exercising a web app with a
+  test browser, we need to exercise a test browser with... um... a web
+  app? (dbrady notes: Now that we have a contact_app, we could try
+  writing some specs and features to run tourbus against it.)
+
+* Web-Sickle is another internal app, written by Tim Harper, that
+  works "well enough". Until I open-sourced this project, it was a
+  submodule in the app. We wanted to keep TourBus extensions separate
+  from WebSickle itself, so there's a lot of code in Runner that
+  really belongs in WebSickle.
+
+* Documentation is <strike>horrible</strike> merely quite bad.
+
+* There's not much in the way of examples, either. When I removed all
+  the LMP-specific code, all of the examples went with it. Sorry about
+  that. Coming soon.
+
+== Feature Requests (How You Can Help!)
+
+* I'd like to beef up the example contact app to show more of TourBus
+  than the simplest possible path. Adding in another page or two, and
+  then adding an additional tour or two would make it more apparent to
+  new users that you can do things like run multiple tours at once.
+  Also, having more than one test_ method in simple.rb would let us
+  demonstrate test filtering as well. (Be aware that at present
+  [2009-04-17], webrat still does not play well with Sinatra sessions
+  so there would be complications. dbrady's fork of webrat combines
+  jferris' fix with the latest webrat, and will be maintained until
+  main webrat includes the feature. That fork is at
+  http://github.com/dbrady/webrat)
+
+* I'd like to remove WebSickle and replace it with Webrat. There is a
+  webrat branch on the main fork (http://github.com/dbrady/tourbus)
+  that is 90% complete. Once that's done we can start massaging the
+  API to be a little more friendly.
+
+== Credits
+
+* Tim Harper camped at my place for a day fixing bugs in WebSickle as
+  I exercised more and more new bits of it. Thanks, dude.
+
+* Lead Media Partners paid me to write TourBus, then let me open
+  source it. How much do they rock? All the way to 11, that's how much
+  they rock.
+
+* James Britt jumped on this and revived it as it was gathering dust.
+  Thanks!
+
+* JT Zemp added before_tour, after_tour. Thanks!
+
+== License
+
+MIT. See the license file.
+

data/bin/tourbus

@@ -0,0 +1,46 @@
+#!/usr/bin/env ruby
+require File.expand_path(File.join(File.dirname(__FILE__), '..', 'lib', 'common'))
+require 'trollop'
+require_all_files_in_folder 'tours'
+
+# load config file, we'll use these as defaults
+config_file = ["./tourbus.yml", "./tours/tourbus.yml", "./config/tourbus.yml", "~/tourbus.yml"].map {|p| File.expand_path(p)}.find {|p| File.exists? p}
+config = config_file ? YAML::load_file(config_file).symbolize_keys : {}
+
+config_map = { :host => :to_s, :concurrency => :to_i, :number => :to_i, :rand => :to_i, :tests => :to_s }
+config_map.each {|key,conv| config[key] = config[key].send(conv) if config.key? key }
+
+# defaults
+config[:host] ||= "localhost:3000"
+config[:concurrency] ||= 1
+config[:number] ||= 1
+config[:rand] ||= nil
+
+opts = Trollop.options do
+  opt :host, "Remote hostname to test", :default => config[:host]
+  opt :concurrency, "Number of simultaneous runs to perform", :type => :integer, :default => config[:concurrency]
+  opt :number, "Number of times to run the tour (in each concurrent step, so -c 10 -n 10 will run the tour 100 times)", :type => :integer, :default => config[:number]
+  opt :list, "List tours and runs available. If tours or runs are included, filters the list", :type => :boolean, :default => nil
+  opt :rand, "Random seed", :type => :integer, :default => config[:rand]
+  opt :tests, "Test name(s) filter. The name of the test to run (use --list to see the test names). Use commas, no spaces, for mulitple names",  :type => :string, :default => nil
+end
+
+tours = if ARGV.empty?
+          Tour.tours
+         else
+           ARGV
+         end
+
+srand opts[:rand] || Time.now.to_i
+
+if opts[:list]
+  Tour.tours(ARGV).each do |tour|
+    puts tour
+    puts Tour.tests(tour).map {|test| "  #{test}"}
+  end
+else
+  opts[:tests] = opts[:tests].split(',') if opts[:tests]
+  
+  TourBus.new(opts[:host], opts[:concurrency], opts[:number], tours, opts[:tests]).run
+end
+

data/bin/tourwatch

@@ -0,0 +1,52 @@
+#!/usr/bin/env ruby
+
+# tourwatch - cheap monitor program for tourbus
+# 
+# Notes:
+# 
+# tourwatch is a cheap logger program for tourbus. It runs on the
+# targeted server and monitors cpu and memory usage of webserver
+# processes. It's a moderately quick hack: I have a 2-hour budget to
+# write and debug the whole thing and here I am wasting time by
+# starting with documentation. This is because I figure the chance of
+# this program needing maintenance in the next 6 months to be well
+# over 100%, and the poor guy behind me (Hey, that's you! Hi.) will
+# need to know why tourwatch is so barebones.
+# 
+# So. TourWatch runs on the target server, collects top information
+# every second, and logs it to file. End of story. "Automation" is
+# handled by the meat cloud (Hey, that's you! Hi.) when the maintainer
+# starts and stops the process manually. Report collection is handled
+# by you reading the logfiles in a terminal. Report aggregation is
+# handled by you aggregating the reports. Yes, there's a theme here.
+# 
+# TODO:
+# 
+# - Remote reporting? Send log events to main log server?
+# 
+# - If we logged to a lightweight database like sqlite3, we could do
+#   some clever things like track individual pids and process groups.
+#   This would let us track, e.g., aggregate apache stress as well as
+#   rogue mongrels. I'm not doing this now because it will require
+#   writing something to read and parse the previous information. For
+#   now, we'll leave it up to the user (Hey, that's you! Hi.) to parse
+#   the logfiles.
+# 
+# - Tweak output format. Currently it's crap. I don't think we need
+#   dynamic templating or anything, but it might be nice to improve
+#   the existing formats.
+
+require File.expand_path(File.join(File.dirname(__FILE__), '..', 'lib', 'common'))
+require 'trollop'
+require 'tour_watch'
+
+opts = Trollop.options do
+  opt :outfile, "Logfile name (default to STDOUT)", :type => :string, :default => nil
+  opt :processes, "csv of processes to monitor", :type => :string, :default => nil
+  opt :cores, "number of cores present (max CPU% is number of cores * 100)", :type => :integer, :default => 4
+  opt :mac, "Set if running on MacOSX. The Mac top command is different than linux top.", :type => :boolean, :default => false
+end
+
+TourWatch.new(opts).run
+
+

data/examples/contact_app/README.rdoc

@@ -0,0 +1,134 @@
+= Contact App
+
+Silly little contact app to show you how to tour a website.
+
+= Requirements
+
+In addition to tourbus, you will need Sinatra to run
+this app.
+
+  sudo gem install sinatra
+
+= Contact App
+
+== Start the app
+
+Once that's working, start the app with "ruby contact_app.rb". Sinatra
+should start up, and you can now point your browser at
+http://localhost:4567 to see the app's homepage.
+
+Pretty humble, I know; just the one link labeled Enter Contacts. Click
+it to get to the Contact form. Here you can enter a first and last
+name then click submit.
+
+The app then shows you that name in last_name, first_name format.
+That's the whole app. Don't everybody applaud all at once.
+
+== First Tour
+
+Still here? Okay, let's tour this website.
+
+In the tours folder, you will find two files: simple.rb and
+tourbus.yml. The YAML file just sets the default host to
+localhost:4567. (Without it, tourbus will default to localhost:3000.
+You could override this by running tourbus with "-h localhost:4567"
+every time, but that gets tedious.
+
+Before we go any farther, let's run tourbus. Leave Sinatra running and
+open another terminal window. Go into the contact_app folder and just
+type "tourbus". You should get a screenful of information ending with
+a happy little banner something like this:
+
+  2009-01-10 12:09:36 TourBus: --------------------------------------------------------------------------------
+  2009-01-10 12:09:36 TourBus: 1 runs: 1x1 of simple
+  2009-01-10 12:09:36 TourBus: All Runners finished.
+  2009-01-10 12:09:36 TourBus: Total Runs: 1
+  2009-01-10 12:09:36 TourBus: Total Passes: 1
+  2009-01-10 12:09:36 TourBus: Total Fails: 0
+  2009-01-10 12:09:36 TourBus: Total Errors: 0
+  2009-01-10 12:09:36 TourBus: Elapsed Time: 0.0131220817565918
+  2009-01-10 12:09:36 TourBus: Speed: 76.207 v/s
+  2009-01-10 12:09:36 TourBus: --------------------------------------------------------------------------------
+
+== Tourbus Defaults
+
+Tourbus tries to be sensible; if you don't provide a number of runs or
+concurrency, it sets them to 1. If you don't choose a tour to run, it
+runs them all. It looks for tourbus.yml in the current folder,
+./tours, in ./config (a Rails convention), and in your home folder.
+(It looks for them in that order, and stops as soon as it finds one.
+It does not merge multiple yaml files together.)
+
+== Simple Tour
+
+Okay, now let's look at tours/simple.rb.
+
+It defines a class named Simple that inherits from Tour. Tourbus won't
+try to run a tour unless the file contains a Tour child class of the
+same name as the file.
+
+Inside the class, methods whose names begin with test_ will
+automatically be run as part of the tour. They are not run in any
+particular order.
+
+=== test_home
+
+Right. Let's look test_home first, because it's simpler:
+
+  def test_home
+    open_site_page "/"
+    click_link :text => /Enter Contact/
+    assert_page_uri_matches "/contacts"
+  end
+
++open_site_page+ is defined in Tour.rb, it opens the given path on the
+host that tourbus is testing.
+
++click_link+ does what you'd expect. It takes a hash that identifies
+the link to click. In this case I chose to identify the link with a
+regexp describing its text label. +click_link+ will raise an exception
+if it cannot find the link to click.
+
++assert_page_uri_matches+ will raise an exception unless the uri
+matches the given string or regexp. If I had passed in a regexp, it
+would have passed if the regexp matched. *Note:* Strings only match at
+the /end/ of the uri; simple containment is not enough. Passing
+"/contacts" works the same as passing %r{/contacts$}.
+
+Clear as mud? "/contacts" would match
+http://localhost:4567/users/42/contacts but not
+http://localhost:4567/contacts/42.
+
+
+=== test_contacts
+
+Okay, let's actually submit a form.
+
+  def test_contacts
+    open_site_page "contacts"
+    submit_form(
+                :identified_by => { :action => %r{/contacts} },
+                :values => {
+                  'first_name' => "Joe",
+                  'last_name' => "Tester"
+                }
+                )
+    assert_page_uri_matches "/contacts"
+    assert_page_body_contains "Tester, Joe"
+  end
+
+test_contacts starts by going directly to the contacts app. Note that
+the leading "/" is optional.
+
++submit_form+ does what its name implies. It finds the correct form to
+submit by matching the action to a regexp, then it sets the form
+values and submits the form. *Note:* Like +click_link+, +submit_form+
+contains some implicit assertions. It actually reads the form looking
+for the named inputs and will raise an exception if any are missing.
+This means you cannot use submit_form to do a blind post to a
+webserver.
+
++assert_page_uri_matches+ we've already seen;
++assert_page_body_contains+ searches the page body for the given text
+or regexp.
+

data/examples/contact_app/contact_app.rb

@@ -0,0 +1,36 @@
+#!/usr/bin/env ruby -w
+
+# Contact app. Example Sinatra application that you can use to test tourbus.
+# 
+# Pretty simple applet. You go to / and enter your contact
+# information. When you click submit, it shows you your name in all
+# caps. Okay, "pretty simple" was an understatement. I get that. Shut up.
+require 'rubygems'
+require 'sinatra'
+
+get '/' do
+  '<a href="/contacts">Enter Contact</a>'
+end 
+
+get '/contacts' do
+  <<-eos
+<html>
+  <head>
+    <title>Contact App</title>
+  </head>
+  <body>
+    <h1>Contact Info:</h1>
+    <form action="/contacts" method="POST">
+      <p><label for="first_name"><b>First Name:</b></label> <input name="first_name" size="30"></p>
+      <p><label for="last_name"><b>Last Name:</b></label> <input name="last_name" size="30"></p>
+      <input type="submit">
+    </form>
+  </body>
+</html>
+eos
+end
+
+post '/contacts' do
+  "<h1>#{params[:last_name]}, #{params[:first_name]}</h1>"
+end 
+

data/examples/contact_app/tours/simple.rb

@@ -0,0 +1,20 @@
+class Simple < Tour
+  def test_home
+    open_site_page "/"
+    click_link :text => /Enter Contact/
+    assert_page_uri_matches "/contacts"
+  end
+
+  def test_contacts
+    open_site_page "contacts"
+    submit_form(
+                :identified_by => { :action => %r{/contacts} },
+                :values => {
+                  'first_name' => "Joe",
+                  'last_name' => "Tester"
+                }
+                )
+    assert_page_uri_matches "/contacts"
+    assert_page_body_contains "Tester, Joe"
+  end
+end

data/examples/contact_app/tours/tourbus.yml

@@ -0,0 +1,2 @@
+---
+host:   localhost

data/lib/common.rb

@@ -0,0 +1,31 @@
+# common.rb - Common settings, requires and helpers
+unless defined? TOURBUS_LIB_PATH
+  TOURBUS_LIB_PATH = File.expand_path(File.join(File.dirname(__FILE__), '..', 'lib'))
+  $:<< TOURBUS_LIB_PATH unless $:.include? TOURBUS_LIB_PATH
+end
+
+require 'rubygems'
+
+gem 'mechanize', ">= 0.8.5"
+gem 'trollop', ">= 1.10.0"
+gem 'faker', '>= 0.3.1'
+
+# TODO: I'd like to remove dependency on Rails. Need to see what all
+# we're using (like classify) and remove each dependency individually.
+require 'activesupport'
+
+require 'monitor'
+require 'faker'
+require 'web-sickle/init'
+require 'tour_bus'
+require 'runner'
+require 'tour'
+
+class TourBusException < Exception; end
+
+def require_all_files_in_folder(folder, extension = "*.rb")
+  for file in Dir[File.join('.', folder, "**/#{extension}")]
+    require file
+  end
+end
+

data/lib/runner.rb

@@ -0,0 +1,72 @@
+require 'monitor'
+require 'common'
+
+class Runner
+  attr_reader :host, :tours, :number, :runner_type, :runner_id
+  
+  def initialize(host, tours, number, runner_id, test_list)
+    @host, @tours, @number, @runner_id, @test_list = host, tours, number, runner_id, test_list
+    @runner_type = self.send(:class).to_s
+    log("Ready to run #{@runner_type}")
+  end
+  
+  # Dispatches to subclass run method
+  def run_tours 
+    log "Filtering on tests #{@test_list.join(', ')}" unless @test_list.to_a.empty?
+    tours,tests,passes,fails,errors = 0,0,0,0,0
+    1.upto(number) do |num|
+      log("Starting #{@runner_type} run #{num}/#{number}")
+      @tours.each do |tour_name|
+        
+        log("Starting run #{number} of Tour #{tour_name}")
+        tours += 1
+        tour = Tour.make_tour(tour_name,@host,@tours,@number,@runner_id)
+        tour.before_tour
+        
+        tour.tests.each do |test|
+
+          next if test_limited_to(test) #  test_list && !test_list.empty? && !test_list.include?(test.to_s) 
+
+          begin
+            tests += 1
+            tour.run_test test
+            passes += 1
+          rescue TourBusException, WebsickleException => e
+            log("********** FAILURE IN RUN! **********")
+            log e.message
+            e.backtrace.each do |trace|
+              log trace
+            end
+            fails += 1
+          rescue Exception => e
+            log("*************************************")
+            log("*********** ERROR IN RUN! ***********")
+            log("*************************************")
+            log e.message
+            e.backtrace.each do |trace|
+              log trace
+            end
+            errors += 1
+          end 
+          log("Finished run #{number} of Tour #{tour_name}")
+        end
+        
+        tour.after_tour
+      end
+      log("Finished #{@runner_type} run #{num}/#{number}")
+    end
+    log("Finished all #{@runner_type} tours.")
+    [tours,tests,passes,fails,errors]
+  end
+  
+  protected
+  
+  def log(message)
+    puts "#{Time.now.strftime('%F %H:%M:%S')} Runner ##{@runner_id}: #{message}"
+  end
+
+  def test_limited_to(test_name)
+    @test_list && !@test_list.empty? && !@test_list.include?(test_name.to_s) 
+  end
+end
+