dbrady-tourbus 0.0.2

data/MIT-LICENSE

@@ -0,0 +1,22 @@
+Copyright (c) 2008-2009 David Brady github@shinybit.com
+ 
+Permission is hereby granted, free of charge, to any person
+obtaining a copy of this software and associated documentation
+files (the "Software"), to deal in the Software without
+restriction, including without limitation the rights to use,
+copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the
+Software is furnished to do so, subject to the following
+conditions:
+ 
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+ 
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.

data/README.txt

@@ -0,0 +1,139 @@
+= TourBus
+
+Flexible and scalable website testing tool.
+
+== Authors
+
+* David Brady -- david.brady@leadmediapartners.com
+* Tim Harper -- tim.harper@leadmediapartners.com
+
+== General Info
+
+TourBus is an intelligent website load testing tool. Allows for
+complicated testing scenarios including filling out forms, following
+redirects, handling cookies, and following links--all of the things
+you'd normally associate with a regression suite or integration
+testing tool. The difference is that TourBus also scales concurrently,
+and you can perform hundreds of complicated regression tests
+simultaneously in order to thoroughly load test your website.
+
+== Motivation
+
+I started writing TourBus because I needed flexibility and scalability
+in a website testing tool, and the extant tools all provided one but
+not the other. Selenium is ultraflexible but limited to the number of
+browsers you can have open at once, while Apache Bench is powerful and
+fast but limited to simple tests.
+
+TourBus lets you define complicated paths through your website, then
+execute those paths concurrently for stress testing.
+
+== Example
+
+To see TourBus in action, you need to write scripts. For lack of a
+better name, these are called Tours. 
+
+=== Example Tour 
+
+* Make a folder called tours and put a file in it called simple.rb. In
+  it write:
+
+  class Simple < Tour
+    def test_homepage
+      open_page "http://#{@host}/"
+      assert_page_body_contains "My Home Page"
+    end
+  end
+
+* Files in ./tours should have classes that match their names. E.g.
+  "class BigHairyTest < Tour" belongs in ./tours/big_hairy_test.rb
+
+* Think Test::Unit. test_* methods will be found automagically.
+  setup() and teardown() methods will be executed at the appropriate
+  times.
+
+=== Example TourBus Run
+
+tourbus -c 2 -n 3 simple
+
+This will create 2 concurrent Tour runners, each of which will run all
+of the methods in Simple three times.
+
+* You can specify multiple tours.
+
+* If you don't specify a tour, all tours in ./tours will be run.
+
+* tourbus --help will give you more information.
+
+=== Example TourWatch Run
+
+On the webserver, you can type 
+
+tourwatch -c 4
+
+To begin running tourwatch. It's basically a stripped-down version of
+top with cheesy text graphs. (TourWatch's development cycles were
+included in the 2 days for TourBus.)
+
+* The -c option is for the total number of cores on the server. The
+  top app will cheerfully report a process as taking 392% CPU if it is
+  using 98% of four cores. This option is only necessary for making
+  the little text graphs scale correctly.
+
+* You can choose which processes to watch by passing a csv to -p:
+  
+  tourwatch -p ruby,mongrel
+
+  Each process name is a partial regexp, so the above would match
+  mongrel AND mongrel_rails, etc.
+
+* tourwatch --help will give you more information.
+
+== History and Status
+
+TourBus began life as a 2-day throwaway app. It is definitely an app
+whose development provides many opportunities for open-source
+contributors to make improvements. It is chock-full of brutal hacks,
+duplications, oversights, and kludges.
+
+== Hacks, Kludges, Known Issues, and Piles of Steaming Poo
+
+* Mechanize 0.8 doesn't always play well together with TourBus. If you
+  get "connection refused" socket errors, try upgrading to Mechanize
+  0.9.
+
+* JRuby doesn't play well with Nokogiri. I have set the html_parser to
+  use hpricot, which should work around the issue for now.
+
+* There are no specs. Yikes! This is to my eternal shame because I'm
+  sort of a testing freak. Because TourBus *WAS* a testing tool, I
+  didn't put tests on it. I haven't put tests on it yet because I'm
+  not sure how to go about it. Instead of exercising a web app with a
+  test browser, we need to exercise a test browser with... um... a web
+  app?
+
+* Web-Sickle is another internal app, written by Tim Harper, that
+  works "well enough". Until I open-sourced this project, it was a
+  submodule in the app. We wanted to keep TourBus extensions separate
+  from WebSickle itself, so there's a lot of code in Runner that
+  really belongs in WebSickle.
+
+* Documentation is horrible.
+
+* There's not much in the way of examples, either. When I removed all
+  the LMP-specific code, all of the examples went with it. Sorry about
+  that. Coming soon.
+
+== Credits
+
+* Tim Harper camped at my place for a day fixing bugs in WebSickle as
+  I exercised more and more new bits of it. Thanks, dude.
+
+* Lead Media Partners paid me to write TourBus, then let me open
+  source it. How much do they rock? All the way to 11, that's how much
+  they rock.
+
+== License
+
+MIT. See the license file.
+

data/bin/tourbus

@@ -0,0 +1,29 @@
+#!/usr/bin/env ruby
+
+require File.expand_path(File.join(File.dirname(__FILE__), '..', 'lib', 'common'))
+require 'trollop'
+require_all_files_in_folder 'tours'
+
+opts = Trollop.options do
+  opt :host, "Remote hostname to test", :default => "localhost:3000"
+  opt :concurrency, "Number of simultaneous runs to perform", :type => :integer, :default => 1
+  opt :number, "Number of times to run the tour (in each concurrent step, so -c 10 -n 10 will run the tour 100 times)", :type => :integer, :default => 1
+  opt :verbose, "Run in verbose mode", :type => :boolean, :default => false
+  opt :list, "List tours and runs available. If tours or runs are included, filters the list", :type => :boolean, :default => nil
+end
+
+tours = if ARGV.empty?
+           Dir[File.join('.', 'tours', '*.rb')].map {|f| File.basename(f, ".rb")}
+         else
+           ARGV
+         end
+
+if opts[:list]
+  Tour.tours(ARGV).each do |tour|
+    puts tour
+    puts Tour.tests(tour).map {|test| "  #{test}"}
+  end
+else
+  puts Benchmark.measure { TourBus.new(opts[:host], opts[:concurrency], opts[:number], tours).run }
+end
+

data/bin/tourwatch

@@ -0,0 +1,52 @@
+#!/usr/bin/env ruby
+
+# tourwatch - cheap monitor program for tourbus
+# 
+# Notes:
+# 
+# tourwatch is a cheap logger program for tourbus. It runs on the
+# targeted server and monitors cpu and memory usage of webserver
+# processes. It's a moderately quick hack: I have a 2-hour budget to
+# write and debug the whole thing and here I am wasting time by
+# starting with documentation. This is because I figure the chance of
+# this program needing maintenance in the next 6 months to be well
+# over 100%, and the poor guy behind me (Hey, that's you! Hi.) will
+# need to know why tourwatch is so barebones.
+# 
+# So. TourWatch runs on the target server, collects top information
+# every second, and logs it to file. End of story. "Automation" is
+# handled by the meat cloud (Hey, that's you! Hi.) when the maintainer
+# starts and stops the process manually. Report collection is handled
+# by you reading the logfiles in a terminal. Report aggregation is
+# handled by you aggregating the reports. Yes, there's a theme here.
+# 
+# TODO:
+# 
+# - Remote reporting? Send log events to main log server?
+# 
+# - If we logged to a lightweight database like sqlite3, we could do
+#   some clever things like track individual pids and process groups.
+#   This would let us track, e.g., aggregate apache stress as well as
+#   rogue mongrels. I'm not doing this now because it will require
+#   writing something to read and parse the previous information. For
+#   now, we'll leave it up to the user (Hey, that's you! Hi.) to parse
+#   the logfiles.
+# 
+# - Tweak output format. Currently it's crap. I don't think we need
+#   dynamic templating or anything, but it might be nice to improve
+#   the existing formats.
+
+require File.expand_path(File.join(File.dirname(__FILE__), '..', 'lib', 'common'))
+require 'trollop'
+require 'tour_watch'
+
+opts = Trollop.options do
+  opt :outfile, "Logfile name (default to STDOUT)", :type => :string, :default => nil
+  opt :processes, "csv of processes to monitor", :type => :string, :default => nil
+  opt :cores, "number of cores present (max CPU% is number of cores * 100)", :type => :integer, :default => 4
+  opt :mac, "Set if running on MacOSX. The Mac top command is different than linux top.", :type => :boolean, :default => false
+end
+
+TourWatch.new(opts).run
+
+

data/lib/common.rb

@@ -0,0 +1,31 @@
+# common.rb - Common settings, requires and helpers
+unless defined? TOURBUS_LIB_PATH
+  TOURBUS_LIB_PATH = File.expand_path(File.join(File.dirname(__FILE__), '..', 'lib'))
+  $:<< TOURBUS_LIB_PATH unless $:.include? TOURBUS_LIB_PATH
+end
+
+require 'rubygems'
+
+gem 'mechanize', ">= 0.8.5"
+gem 'trollop', ">= 1.10.0"
+gem 'faker', '>= 0.3.1'
+
+# TODO: I'd like to remove dependency on Rails. Need to see what all
+# we're using (like classify) and remove each dependency individually.
+require 'activesupport'
+
+require 'monitor'
+require 'faker'
+require 'web-sickle/init'
+require 'tour_bus'
+require 'runner'
+require 'tour'
+
+
+class TourBusException < Exception; end
+
+def require_all_files_in_folder(folder, extension = "*.rb")
+  for file in Dir[File.join('.', folder, "**/#{extension}")]
+    require file
+  end
+end

data/lib/runner.rb

@@ -0,0 +1,67 @@
+require 'monitor'
+require 'common'
+
+class Runner
+  attr_reader :host, :tours, :number, :runner_type, :runner_id
+  
+  def initialize(host, tours, number, runner_id)
+    @host, @tours, @number, @runner_id = host, tours, number, runner_id
+    @runner_type = self.send(:class).to_s
+    log("Ready to run #{@runner_type}")
+  end
+  
+  # Dispatches to subclass run method
+  def run_tours
+    runs,passes,fails,errors = 0,0,0,0
+    1.upto(number) do |num|
+      log("Starting #{@runner_type} run #{num}/#{number}")
+      begin
+        @tours.each do |tour|
+          runs += 1
+          tour = Tour.make_tour(tour,@host,@tours,@number,@runner_id)
+          tour.tests.each do |test|
+            tour.run_test test
+          end
+        end
+        passes += 1
+      rescue TourBusException => e
+        log("***********************************")
+        log("********** ERROR IN RUN! **********")
+        log("***********************************")
+        log e.message
+        e.backtrace.each do |trace|
+          log trace
+        end
+        fails += 1
+      rescue WebsickleException => e
+        log("***********************************")
+        log("********** ERROR IN RUN! **********")
+        log("***********************************")
+        log e.message
+        e.backtrace.each do |trace|
+          log trace
+        end
+        fails += 1
+      rescue Exception => e
+        log("***********************************")
+        log("********** ERROR IN RUN! **********")
+        log("***********************************")
+        log e.message
+        e.backtrace.each do |trace|
+          log trace
+        end
+        errors += 1
+      end 
+      log("Finished #{@runner_type} run #{num}/#{number}")
+    end
+    log("Finished all #{@runner_type} runs.")
+    [runs,passes,fails,errors]
+  end
+  
+  protected
+  
+  def log(message)
+    puts "#{Time.now.strftime('%F %H:%M:%S')} Runner ##{@runner_id}: #{message}"
+  end
+end
+

data/lib/tour.rb

@@ -0,0 +1,110 @@
+require 'monitor'
+require 'common'
+
+# A tour is essentially a test suite file. A Tour subclass
+# encapsulates a set of tests that can be done, and may contain helper
+# and support methods for a given task. If you have a two or three
+# paths through a specific area of your website, define a tour for
+# that area and create test_ methods for each type of test to be done.
+
+class Tour
+  include WebSickle
+  attr_reader :host, :tours, :number, :tour_type, :tour_id
+  
+  def initialize(host, tours, number, tour_id)
+    @host, @tours, @number, @tour_id = host, tours, number, tour_id
+    @tour_type = self.send(:class).to_s
+  end
+  
+  def setup
+  end
+  
+  def teardown
+  end
+  
+  # Lists tours in tours folder. If a string is given, filters the
+  # list by that string. If an array of filter strings is given,
+  # returns items that match ANY filter string in the array.
+  def self.tours(filter=[])
+    filter = [filter].flatten
+    # All files in tours folder, stripped to basename, that match any item in filter
+    Dir[File.join('.', 'tours', '**', '*.rb')].map {|fn| File.basename(fn, ".rb")}.select {|fn| filter.size.zero? || filter.any?{|f| fn =~ /#{f}/}}
+  end 
+  
+  def self.tests(tour_name)
+    Tour.make_tour.tests
+  end
+  
+  # Factory method, creates the named child class instance
+  def self.make_tour(tour_name,host,tours,number,tour_id)
+    tour_name.classify.constantize.new(host,tours,number,tour_id)
+  end
+  
+  # Returns list of tests in this tour. (Meant to be run on a subclass
+  # instance; returns the list of tests available).
+  def tests
+    methods.grep(/^test_/).map {|m| m.sub(/^test_/,'')}
+  end
+  
+  def run_test(test_name)
+    test = "test_#{test_name}"
+    raise TourBusException.new("run_test couldn't run test '#{test_name}' because this tour did not respond to :#{test}") unless respond_to? test
+    setup
+    send test
+    teardown
+  end
+    
+  protected
+
+  def log(message)
+    puts "#{Time.now.strftime('%F %H:%M:%S')} Tour ##{@tour_id}: #{message}"
+  end
+
+  # given "portal", opens "http://#{@host}/portal"
+  def open_site_page(path)
+    open_page "http://#{@host}/#{path}"
+  end
+  
+  def dump_form
+    log "Dumping Forms:"
+    page.forms.each do |form|
+      puts "Form: #{form.name}"
+      puts '-' * 20
+      (form.fields + form.radiobuttons + form.checkboxes + form.file_uploads).each do |field|
+        puts "  #{field.name}"
+      end
+    end
+  end
+  
+  # True if uri ends with the string given. If a regex is given, it is
+  # matched instead.
+  # 
+  # TODO: Refactor me--these were separated out back when Websickle
+  # was a shared submodule and we couldn't pollute it. Now that it's
+  # frozen these probably belong there.
+  def assert_page_uri_matches(uri)
+    case uri
+    when String:
+        raise WebsickleException, "Expected page uri to match String '#{uri}' but did not. It was #{page.uri}" unless page.uri.to_s[-uri.size..-1] == uri
+    when Regexp:
+        raise WebsickleException, "Expected page uri to match Regexp '#{uri}' but did not. It was #{page.uri}" unless page.uri.to_s =~ uri
+    end
+    log "Page URI ok (#{page.uri} matches: #{uri})"
+  end
+  
+  # True if page contains (or matches) the given string (or regexp)
+  # 
+  # TODO: Refactor me--these were separated out back when Websickle
+  # was a shared submodule and we couldn't pollute it. Now that it's
+  # frozen these probably belong there.
+  def assert_page_body_contains(pattern)
+    case pattern
+    when String:
+        raise WebsickleException, "Expected page body to contain String '#{pattern}' but did not. It was #{page.body}" unless page.body.to_s.index(pattern)
+    when Regexp:
+        raise WebsickleException, "Expected page body to match Regexp '#{pattern}' but did not. It was #{page.body}" unless page.body.to_s =~ pattern
+    end
+    log "Page body ok (matches #{pattern})"
+  end
+end
+

data/lib/tour_bus.rb

@@ -0,0 +1,84 @@
+require 'benchmark'
+
+class TourBus < Monitor
+  attr_reader :host, :concurrency, :number, :tours, :runs, :passes, :fails, :errors, :benchmarks
+  
+  def initialize(host="localhost", concurrency=1, number=1, tours=[])
+    @host, @concurrency, @number, @tours = host, concurrency, number, tours
+    @runner_id = 0
+    @runs, @passes, @fails, @errors = 0,0,0,0
+    super()
+  end
+  
+  def next_runner_id
+    synchronize do
+      @runner_id += 1
+    end 
+  end
+  
+  def update_stats(runs,passes,fails,errors)
+    synchronize do
+      @runs += runs
+      @passes += passes
+      @fails += fails
+      @errors += errors
+    end
+  end
+  
+  def update_benchmarks(bm)
+    synchronize do
+      @benchmarks = @benchmarks.zip(bm).map { |a,b| a+b}
+    end 
+  end
+  
+  def runners(filter=[])
+    # All files in tours folder, stripped to basename, that match any item in filter
+    Dir[File.join('.', 'tours', '**', '*.rb')].map {|fn| File.basename(fn, ".rb")}.select {|fn| filter.size.zero? || filter.any?{|f| fn =~ /#{f}/}}
+  end
+  
+  def run
+    threads = []
+    started = Time.now.to_f
+    concurrency.times do |conc|
+      log "Starting #{concurrency} runners to run #{tours.size} tours #{number} times (for a total of #{tours.size*concurrency*number} times)"
+      threads << Thread.new do
+        runner_id = next_runner_id
+        runs,passes,fails,errors,start = 0,0,0,0,Time.now.to_f
+        bm = Benchmark.measure do
+          runner = Runner.new(@host, @tours, @number, runner_id)
+          runs,passes,fails,errors = runner.run_tours
+          update_stats runs, passes, fails, errors
+        end
+        log "Runner Finished!"
+        log "Runner finished in %0.3f seconds" % (Time.now.to_f - start)
+        log "Runner Finished! runs,passes,fails,errors: #{runs},#{passes},#{fails},#{errors}"
+        log "Benchmark for runner #{runner_id}: #{bm}"
+      end
+    end
+    log "All Runners started!"
+    threads.each {|t| t.join }
+    finished = Time.now.to_f
+    log '-' * 80
+    log "All Runners finished."
+    log "Total Runs: #{@runs}"
+    log "Total Passes: #{@passes}"
+    log "Total Fails: #{@fails}"
+    log "Total Errors: #{@errors}"
+    log "Elapsed Time: #{finished - started}"
+    log "Speed: %5.3f v/s" % (@runs / (finished-started))
+    log '-' * 80
+    if @fails > 0 || @errors > 0
+      log '********************************************************************************'
+      log '********************************************************************************'
+      log '                            !! THERE WERE FAILURES !!'
+      log '********************************************************************************'
+      log '********************************************************************************'
+    end
+  end
+  
+  def log(message)
+    puts "#{Time.now.strftime('%F %H:%M:%S')} TourBus: #{message}"
+  end
+
+end
+

data/lib/tour_watch.rb

@@ -0,0 +1,88 @@
+class TourWatch
+  attr_reader :processes
+  
+  def initialize(options={})
+    @processes = if options[:processes]
+                   options[:processes].split(/,/) * '|'
+                 else 
+                   "ruby|mysql|apache|http|rails|mongrel"
+                 end
+    @cores = options[:cores] || 4
+    @logfile = options[:outfile]
+    @mac = options[:mac]
+  end
+  
+  def stats
+    top = @mac ? top_mac : top_linux
+    lines = []
+    @longest = Hash.new(0)
+    top.each_line do |line|
+      name,pid,cpu = fields(line.split(/\s+/))
+      lines << [name,pid,cpu]
+      @longest[:name] = name.size if name.size > @longest[:name]
+      @longest[:pid] = pid.to_s.size if pid.to_s.size > @longest[:pid]
+    end
+    lines
+  end
+  
+  def fields(parts)
+    @mac ? fields_mac(parts) : fields_linux(parts)
+  end
+  
+  # Note: MacOSX is so awesome I just cacked. Top will report 0.0% cpu
+  # the first time you run top, every time. The only way to get actual
+  # CPU% here is to wait for it to send another page and then throw
+  # away the first page. Isn't that just awesome?!? I KNOW!!!
+  def top_mac
+    top = `top -l 1 | grep -E '(#{@processes})'`
+  end
+  
+  def fields_mac(fields)
+    name,pid,cpu = fields[1], fields[0].to_i, fields[2].to_f
+  end
+  
+  def top_linux
+    top = `top -bn 1 | grep -E '(#{@processes})'`
+  end
+  
+  
+  def fields_linux(fields)
+    # linux top isn't much smarter. It spits out a blank field ahead
+    # of the pid if the pid is too short, which makes the indexes
+    # shift off by one.
+    a,b,c = if fields.size == 13
+              [-1,1,9]
+            else
+              [-1,0,8]
+            end
+    name,pid,cpu = fields[a], fields[b].to_i, fields[c].to_f
+  end
+  
+  
+  def run()
+    while(true)
+      now = Time.now.to_i
+      if @time != now
+        log '--'
+        lines = stats
+        lines.sort! {|a,b| a[1]==b[1] ? a[2]<=>b[2] : a[1]<=>b[1] }
+        lines.each do |vars|
+          vars << bargraph(vars[2], 100 * @cores)
+          log "%#{@longest[:name]}s %#{@longest[:pid]}d CPU: %6.2f%% [%-40s]" % vars 
+        end
+      end
+      sleep 0.1
+      @time = now
+    end
+  end
+  
+  def bargraph(value, max=100, length=40, on='#', off='.')
+    (on * (([[value, 0].max, max].min * length) / max).to_i).ljust(length, off)
+  end
+  
+  def log(message)
+    msg = "#{Time.now.strftime('%F %H:%M:%S')} TourWatch: #{message}"
+    puts msg
+    File.open(@logfile, "a") {|f| f.puts msg } if @logfile
+  end
+end

data/lib/web-sickle/init.rb

@@ -0,0 +1,17 @@
+require 'rubygems'
+gem 'mechanize', ">= 0.7.6"
+gem "hpricot", ">= 0.6"
+$: << File.join(File.dirname(__FILE__), 'lib')
+
+require 'hpricot'
+require 'mechanize'
+
+WWW::Mechanize.html_parser = Hpricot
+
+require 'web_sickle'
+require "assertions"
+require "hash_proxy"
+require "helpers/asp_net"
+require "helpers/table_reader"
+
+Hpricot.buffer_size = 524288

data/lib/web-sickle/lib/assertions.rb

@@ -0,0 +1,51 @@
+class WebSickleAssertionException < Exception; end
+
+module WebSickle::Assertions
+  def assert_equals(expected, actual, message = nil)
+    unless(expected == actual)
+      report_error <<-EOF
+Error: Expected 
+#{expected.inspect}, but got
+#{actual.inspect}
+#{message}
+EOF
+    end
+  end
+
+  def assert_select(selector, message)
+    assert_select_in(@page, selector, message)
+  end
+
+  def assert_no_select(selector, message)
+    assert_no_select_in(@page, selector, message)
+  end
+
+  def assert_select_in(content, selector, message)
+    report_error("Error: Expected selector #{selector.inspect} to find a page element, but didn't. #{message}") if (content / selector).blank?
+  end
+
+  def assert_no_select_in(content, selector, message)
+    report_error("Error: Expected selector #{selector.inspect} to not find a page element, but did. #{message}") unless (content / selector).blank?
+  end
+
+  def assert_contains(left, right, message = nil)
+    (right.is_a?(Array) ? right : [right]).each do | item |
+      report_error("Error: Expected #{left.inspect} to contain #{right.inspect}, but didn't. #{message}") unless left.include?(item)
+    end
+  end
+
+  def assert(passes, message = nil)
+    report_error("Error: expected true, got false.  #{message}") unless passes
+  end
+
+  def assert_link_text(link, text)
+    case text
+    when String
+      assert_equals(link.text, text)
+    when Regexp
+      assert(link.text.match(text))
+    else
+      raise ArgumentError, "Don't know how to assert an object like #{text.inspect} - expected: Regexp or String"
+    end
+  end
+end

data/lib/web-sickle/lib/hash_proxy.rb

@@ -0,0 +1,9 @@
+class HashProxy
+  def initialize(options = {})
+    @set = options[:set]
+    @get = options[:get]
+  end
+
+  def [](key); @get && @get.call(key); end
+  def []=(key, value); @set && @set.call(key, value); end
+end

data/lib/web-sickle/lib/helpers/asp_net.rb

@@ -0,0 +1,16 @@
+module WebSickle::Helpers
+module AspNet
+  def asp_net_do_postback(options)
+    target_element = case
+      when options[:button]
+        find_button(options[:button])
+      when options[:field]
+        find_field(options[:field])
+      else
+        nil
+      end
+    @form.fields << WWW::Mechanize::Form::Field.new("__EVENTTARGET", target_element ? target_element.name : "") if target_element
+    submit_form_button
+  end
+end
+end

data/lib/web-sickle/lib/helpers/table_reader.rb

@@ -0,0 +1,39 @@
+module WebSickle::Helpers
+class TableReader
+  attr_reader :headers, :options, :body_rows, :header_row, :extra_rows
+  
+  def initialize(element, p_options = {})
+    @options = {
+      :row_selectors => [" > tr", "thead > tr", "tbody > tr"],
+      :header_selector => " > th",
+      :header_proc => lambda { |th| th.inner_text.gsub(/[\n\s]+/, ' ').strip },
+      :body_selector => " > td",
+      :body_proc => lambda { |header, td| td.inner_text.strip },
+      :header_offset => 0,
+      :body_offset => 1
+    }.merge(p_options)
+    @options[:body_range] ||= options[:body_offset]..-1
+    raw_rows = options[:row_selectors].map{|row_selector| element / row_selector}.compact.flatten
+    
+    @header_row = raw_rows[options[:header_offset]]
+    @body_rows = raw_rows[options[:body_range]]
+    @extra_rows = (options[:body_range].last+1)==0 ? [] : raw_rows[(options[:body_range].last+1)..-1]
+    
+    @headers = (@header_row / options[:header_selector]).map(&options[:header_proc])
+  end
+  
+  def rows
+    @rows ||= @body_rows.map do |row|
+      hash = {}
+      data_array = (headers).zip(row / options[:body_selector]).each do |column_name, td|
+        hash[column_name] = options[:body_proc].call(column_name, td)
+      end
+      hash
+    end
+  end
+  
+  def array_to_hash(data, column_names)
+    column_names.inject({}) {|h,column_name| h[column_name] = data[column_names.index(column_name)]; h }
+  end
+end
+end

data/lib/web-sickle/lib/make_nokigiri_output_useful.rb

@@ -0,0 +1,15 @@
+# Nokogiri::XML::Element.class_eval do
+#   def inspect(indent = "")
+#     breaker = "\n#{indent}"
+#     if children.length == 0
+#       %(#{indent}<#{name}#{breaker}  #{attributes.map {|k,v| k + '=' + v.inspect} * "#{breaker}  "}/>)
+#     else
+#       %(#{indent}<#{name} #{attributes.map {|k,v| k + '=' + v.inspect} * " "}>\n#{children.map {|c| c.inspect(indent + '  ') rescue c.class} * "\n"}#{breaker}</#{name}>)
+#     end
+#   end
+# end
+# Nokogiri::XML::Text.class_eval do
+#   def inspect(indent = "")
+#     "#{indent}#{text.inspect}"
+#   end
+# end

data/lib/web-sickle/lib/web_sickle.rb

@@ -0,0 +1,224 @@
+class WebsickleException < Exception; end
+
+module WebSickle
+  # form_value is used to interface with the current select form
+  attr_reader :form_value
+  attr_accessor :page
+  
+  def initialize(options = {})
+    @page = nil
+    @form_value = HashProxy.new(
+      :set => lambda { |identifier, value| set_form_value(identifier, value)}, 
+      :get => lambda { |identifier| get_form_value(identifier)}
+    )
+  end
+  
+  def click_link(link)
+    set_page(agent.click(find_link(link)))
+  end
+  
+  def submit_form(options = {})
+    options[:button] = :first unless options.has_key?(:button)
+    options[:identified_by] ||= :first
+    select_form(options[:identified_by])
+    set_form_values(options[:values]) if options[:values]
+    submit_form_button(options[:button])
+  end
+
+  # select the current form
+  def select_form(identifier = {})
+    identifier = make_identifier(identifier, [:name, :action, :method])
+    @form = find_in_collection(@page.forms, identifier)
+    report_error("Couldn't find form on page at #{@page.uri} with attributes #{identifier.inspect}") if @form.nil?
+    @form
+  end
+  
+  # submits the current form
+  def submit_form_button(button_criteria = nil, options = {})
+    button = 
+      case button_criteria
+      when nil
+        nil
+      else
+        find_button(button_criteria)
+      end
+    set_page(agent.submit(@form, button))
+  end
+
+  # sets the given path to the current page, then opens it using our agent
+  def open_page(path, parameters = [], referer = nil)
+    set_page(agent.get(path, parameters, referer))
+  end
+
+  # uses Hpricot style css selectors to find the elements in the current +page+.
+  # Uses Hpricot#/ (or Hpricot#search)
+  def select_element(match)
+    select_element_in(@page, match)
+  end
+  
+  # uses Hpricot style css selectors to find the element in the given container.  Works with html pages, and file pages that happen to have xml-like content.
+  # throws error if can't find a match
+  def select_element_in(contents, match)
+    result = (contents.respond_to?(:/) ? contents : Hpricot(contents.body)) / match
+    if result.blank?
+      report_error("Tried to find element matching #{match}, but couldn't")
+    else
+      result
+    end
+  end
+
+  # uses Hpricot style css selectors to find the element.  Works with html pages, and file pages that happen to have xml-like content.
+  # throws error if can't find a match
+  # Uses Hpricot#at
+  def detect_element(match)
+    result = (@page.respond_to?(:at) ? @page : Hpricot(@page.body)).at(match)
+    if result.blank?
+      report_error("Tried to find element matching #{match}, but couldn't")
+    else
+      result
+    end
+  end
+  
+  protected
+    # our friendly mechinze agent
+    def agent
+      @agent ||= new_mechanize_agent
+    end
+    
+    def make_identifier(identifier, valid_keys = nil, default_key = :name)
+      identifier = {default_key => identifier} unless identifier.is_a?(Hash) || identifier.is_a?(Symbol)
+      identifier.assert_valid_keys(valid_keys) if identifier.is_a?(Hash) && valid_keys
+      identifier
+    end
+  
+    def find_field(identifier)
+      if @form.nil?
+        report_error("No form is selected when trying to find field by #{identifier.inspect}")
+        return
+      end
+      identifier = make_identifier(identifier, [:name, :value])
+      find_in_collection(@form.radiobuttons + @form.fields + @form.checkboxes + @form.file_uploads, identifier) ||
+        report_error("Tried to find field identified by #{identifier.inspect}, but failed.\nForm fields are: #{(@form.radiobuttons + @form.fields + @form.checkboxes + @form.file_uploads).map{|f| f.inspect} * ", \n  "}") 
+    end
+    
+    def find_link(identifier)
+      identifier = make_identifier(identifier, [:href, :text], :text)
+      find_in_collection(page.links, identifier) ||
+        report_error("Tried to find link identified by #{identifier.inspect}, but failed.\nValid links are: #{page.links.map{|f| f.inspect} * ", \n  "}")
+    end
+  
+    # finds a button by parameters.  Throws error if not able to find.
+    # example:
+    # find_button("btnSubmit") - finds a button named "btnSubmit"
+    # find_button(:name => "btnSubmit")
+    # find_button(:name => "btnSubmit", :value => /Lucky/) - finds a button named btnSubmit with a value matching /Lucky/
+    def find_button(identifier)
+      identifier = make_identifier(identifier, [:value, :name])
+      find_in_collection(@form.buttons, identifier) ||
+        report_error("Tried to find button identified by #{identifier.inspect}, but failed.  Buttons on selected form are: #{@form.buttons.map{|f| f.inspect} * ','}")
+    end
+  
+    # the magic method that powers find_button, find_field.  Does not throw an error if not found
+    def find_in_collection(collection, identifier, via = :find)
+      return collection.first if identifier == :first
+      find_all_in_collection(collection, identifier, :find)
+    end
+    
+    def find_all_in_collection(collection, identifier, via = :select)
+      return [collection.first] if identifier == :first
+      collection.send(via) do |item|
+        identifier.all? { |k, criteria| is_a_match?(criteria, item.send(k)) }
+      end
+    end
+
+    # sets a form-field's value by identifier.  Throw's error if field does not exist
+    def set_form_value(identifier, value)
+      field = find_field(identifier)
+      case field
+      when WWW::Mechanize::Form::CheckBox
+        field.checked = value
+      when WWW::Mechanize::Form::RadioButton
+        radio_collection = find_all_in_collection(@form.radiobuttons, :name => field.name)
+        radio_collection.each { |f|; f.checked = false }
+        finder = (value.is_a?(Hash) || value.is_a?(Symbol)) ? value : {:value => value}
+        find_in_collection(radio_collection, finder).checked = true
+      when WWW::Mechanize::Form::SelectList
+        if value.is_a?(Hash) || value.is_a?(Symbol)
+          field.value = find_in_collection(field.options, value).value
+        else
+          field.value = value
+        end
+      else
+        field.value = value
+      end
+    end
+  
+    def set_form_values(set_pairs = {})
+      flattened_value_hash(set_pairs).each do |identifier, value|
+        set_form_value(identifier, value)
+      end
+    end
+    
+    def flattened_value_hash(hash, parents = [])
+      new_hash = {}
+      hash.each do |key, value|
+        if value.is_a?(Hash) && value.keys.first.is_a?(String)
+          new_hash.update(flattened_value_hash(value, [key] + parents))
+        else
+          parents.each { |parent| key = "#{parent}[#{key}]"}
+          new_hash[key] = value
+        end
+      end
+      new_hash
+    end
+
+    # sets a form-field's value by identifier.  Throw's error if field does not exist
+    def get_form_value(identifier)
+      field = find_field(identifier)
+      case field
+      when WWW::Mechanize::Form::CheckBox
+        field.checked
+      else
+        field.value
+      end
+    end
+    
+    def format_error(msg)
+      error = "Error encountered: #{msg}."
+      begin
+        error << "\n\nPage URL:#{@page.uri.to_s}" if @page
+      rescue
+      end
+      error
+    end
+
+    def report_error(msg)
+      raise WebsickleException, format_error(msg)
+      nil
+    end
+    
+  private
+    def set_page(p)
+      @form = nil
+      @page = p
+    end
+
+    def is_a_match?(criteria, value)
+      case criteria
+      when Regexp
+        criteria.match(value)
+      when String
+        criteria == value
+      when Array
+        criteria.include?(value)
+      else
+        criteria.to_s == value.to_s
+      end
+    end
+  
+    def new_mechanize_agent
+      a = WWW::Mechanize.new
+      a.read_timeout = 600 # 10 minutes
+      a
+    end
+end

data/lib/web-sickle/spec/lib/helpers/table_reader_spec.rb

@@ -0,0 +1,137 @@
+require File.dirname(__FILE__) + '/../../spec_helper'
+
+describe WebSickle::Helpers::TableReader do
+  describe "Simple example" do
+    before(:each) do
+      @content = <<-EOF
+        <table>
+          <tr>
+            <th>Name</th>
+            <th>Age</th>
+          </tr>
+          <tr>
+            <td>Googly</td>
+            <td>2</td>
+          </tr>
+        </table>    
+      EOF
+      h = Hpricot(@content)
+      @table = WebSickle::Helpers::TableReader.new(h / "table")
+    end
+  
+    it "should extract headers" do
+      @table.headers.should == ["Name", "Age"]
+    end
+  
+    it "should extract rows" do
+      @table.rows.should == [
+        {"Name" => "Googly", "Age" => "2"}
+      ]
+    end
+  end
+  
+  
+  
+  describe "Targetted example" do
+    before(:each) do
+      @content = <<-EOF
+        <table>
+          <thead>
+            <tr>
+              <td colspan='2'>----</td>
+            </tr>
+            <tr>
+              <th><b>Name</b></th>
+              <th><b>Age</b></th>
+            </tr>
+          </thead>
+          <tbody>  
+            <tr>
+              <td>Googly</td>
+              <td>2</td>
+            </tr>
+            <tr>
+              <td>Bear</td>
+              <td>3</td>
+            </tr>
+            <tr>
+              <td colspan='2'>Totals!</td>
+            </tr>
+            <tr>
+              <td>---</td>
+              <td>5</td>
+            </tr>
+          </tbody>
+        </table>    
+      EOF
+      h = Hpricot(@content)
+      @table = WebSickle::Helpers::TableReader.new(h / " > table",
+        :header_selector => " > th > b",
+        :header_offset => 1,
+        :body_range => 2..-3
+      )
+    end
+    
+    it "should extract the column headers" do
+      @table.headers.should == ["Name", "Age"]
+    end
+    
+    it "should extract the row data for the specified range" do
+      @table.rows.should == [
+        {"Name" => "Googly", "Age" => "2"},
+        {"Name" => "Bear", "Age" => "3"},
+      ]
+    end
+    
+    it "should allow you to check extra rows to assert you didn't chop off too much" do
+      (@table.extra_rows.first / "td").inner_text.should == "Totals!"
+    end
+  end
+  
+  
+  
+  describe "when using procs to extract data" do
+    before(:each) do
+      @content = <<-EOF
+        <table>
+          <tr>
+            <th>Name</th>
+            <th>Age</th>
+          </tr>
+          <tr>
+            <td>Googly</td>
+            <td>2</td>
+          </tr>
+          <tr>
+            <td>Bear</td>
+            <td>3</td>
+          </tr>
+        </table>    
+      EOF
+      h = Hpricot(@content)
+      @table = WebSickle::Helpers::TableReader.new(h / " > table",
+        :header_proc => lambda {|th| th.inner_text.downcase.to_sym},
+        :body_proc => lambda {|col_name, td| 
+          value = td.inner_text
+          case col_name
+          when :name
+            value.upcase
+          when :age
+            value.to_i
+          end
+        }
+      )
+    end
+    
+    it "should use the header proc to extract column headers" do
+      @table.headers.should == [:name, :age]
+    end
+    
+    it "should use the body proc to format the data" do
+      @table.rows.should == [
+        {:name => "GOOGLY", :age => 2},
+        {:name => "BEAR", :age => 3}
+      ]
+    end
+  end
+end

data/lib/web-sickle/spec/spec_helper.rb

@@ -0,0 +1,7 @@
+require File.dirname(__FILE__) + '/../init.rb'
+require 'rubygems'
+require 'hpricot'
+require 'test/unit'
+require 'spec'
+require 'active_support'
+require File.dirname(__FILE__) + '/spec_helpers/mechanize_mock_helper.rb'

data/lib/web-sickle/spec/spec_helpers/mechanize_mock_helper.rb

@@ -0,0 +1,12 @@
+module MechanizeMockHelper
+  def fixture_file(filename)
+    File.read("#{File.dirname(__FILE__)}/../fixtures/#{filename}")
+  end
+  
+  def mechanize_page(path_to_data, options = {})
+    options[:uri] ||= URI.parse("http://url.com/#{path_to_data}")
+    options[:response] ||= {'content-type' => 'text/html'}
+  
+    WWW::Mechanize::Page.new(options[:uri], options[:response], fixture_file("/#{path_to_data}"))
+  end
+end

data/lib/web-sickle/spec/web_sickle_spec.rb

@@ -0,0 +1,50 @@
+require File.dirname(__FILE__) + '/spec_helper'
+
+class WebSickleHelper
+  include WebSickle
+end
+
+describe WebSickle do
+  include MechanizeMockHelper
+  
+  before(:all) do
+    WebSickleHelper.protected_instance_methods.each do |method|
+      WebSickleHelper.send(:public, method)
+    end
+  end
+  
+  before(:each) do
+    @helper = WebSickleHelper.new
+  end
+  
+  it "should flatten a value hash" do
+    @helper.flattened_value_hash("contact" => {"first_name" => "bob"}).should == {"contact[first_name]" => "bob"}
+  end
+  
+  describe "clicking links" do
+    before(:each) do
+      @helper.stub!(:page).and_return(mechanize_page("linkies.html"))
+    end
+    
+    it "should click a link by matching the link text" do
+      @helper.agent.should_receive(:click) do |link|
+        link.text.should include("one")
+      end
+      @helper.click_link(:text => /one/)
+    end
+    
+    it "should click a link by matching the link href" do
+      @helper.agent.should_receive(:click) do |link|
+        link.href.should include("/two")
+      end
+      @helper.click_link(:href => %r{/two})
+    end
+    
+    it "should default matching the link text" do
+      @helper.agent.should_receive(:click) do |link|
+        link.text.should include("Link number one")
+      end
+      @helper.click_link("Link number one")
+    end
+  end
+end

metadata

@@ -0,0 +1,114 @@
+--- !ruby/object:Gem::Specification 
+name: dbrady-tourbus
+version: !ruby/object:Gem::Version 
+  version: 0.0.2
+platform: ruby
+authors: 
+- David Brady
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2009-01-05 00:00:00 -08:00
+default_executable: 
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: mechanize
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: 0.8.5
+    version: 
+- !ruby/object:Gem::Dependency 
+  name: trollop
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: "0"
+    version: 
+- !ruby/object:Gem::Dependency 
+  name: faker
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: "0"
+    version: 
+- !ruby/object:Gem::Dependency 
+  name: hpricot
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: "0"
+    version: 
+description: TourBus web stress-testing tool
+email: github@shinybit.com
+executables: 
+- tourbus
+- tourwatch
+extensions: []
+
+extra_rdoc_files: 
+- README.txt
+- MIT-LICENSE
+files: 
+- bin/tourbus
+- bin/tourwatch
+- lib/common.rb
+- lib/runner.rb
+- lib/tour.rb
+- lib/tour_bus.rb
+- lib/tour_watch.rb
+- lib/web-sickle/init.rb
+- lib/web-sickle/lib/assertions.rb
+- lib/web-sickle/lib/hash_proxy.rb
+- lib/web-sickle/lib/helpers/asp_net.rb
+- lib/web-sickle/lib/helpers/table_reader.rb
+- lib/web-sickle/lib/make_nokigiri_output_useful.rb
+- lib/web-sickle/lib/web_sickle.rb
+- lib/web-sickle/spec/lib/helpers/table_reader_spec.rb
+- lib/web-sickle/spec/spec_helper.rb
+- lib/web-sickle/spec/spec_helpers/mechanize_mock_helper.rb
+- lib/web-sickle/spec/web_sickle_spec.rb
+- README.txt
+- MIT-LICENSE
+has_rdoc: true
+homepage: http://github.com/dbrady/tourbus
+post_install_message: 
+rdoc_options: 
+- --line-numbers
+- --inline-source
+- --main
+- README.txt
+- --title
+- Tourbus - Web Stress Testing in Ruby
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.2.0
+signing_key: 
+specification_version: 2
+summary: TourBus web stress-testing tool
+test_files: []
+