topfunky-bong 0.0.2

data/History.txt

@@ -0,0 +1,12 @@
+== 0.0.2 / ???
+
+* ???
+
+== 0.0.1 / 2007-11-07
+
+* First release. See README.txt for the details.
+
+== TODO
+
+* Compare multiple benchmarking runs and generate a report.
+* Generate Gruff graphs or an HTML report.

data/Manifest.txt

@@ -0,0 +1,6 @@
+History.txt
+Manifest.txt
+README.txt
+Rakefile
+bin/bong
+lib/bong.rb

data/README.txt

@@ -0,0 +1,112 @@
+bong
+    by Geoffrey Grosenbach
+    boss@topfunky.com
+    http://topfunky.com
+
+== DESCRIPTION:
+  
+Hit your website with bong. Uses httperf to run a suite of benchmarking tests against specified urls on your site. 
+
+Graphical output and multi-test comparisons are planned. Apache ab support may be added in the future.
+
+== USAGE
+
+See all options with:
+
+  % bong --help
+
+Generate a config file (writes to config/httperf.yml by default):
+
+  % bong --generate 
+
+Edit the config file with a list of servers, urls, and your preferred sample size. NOTE: Don't run this against servers you don't own!
+
+  --- 
+  uris: 
+  - /
+  - /pages/about
+  samples: 2
+  servers: 
+  - localhost:3000
+
+Run the benchmarking suite, label it 'baseline', and output the raw data to a file (writes output to log/httperf-report.yml by default):
+
+  % bong baseline
+
+A report will be printed to the screen and the raw data will be saved to log/httperf-report.yml (change with the --out option). It's a good idea to use a label for each test so you can compare them later and find out what the fastest implementation was. Examples: 'baseline', 'memcached-optimization', 'sql-queries', etc.
+
+  baseline
+    localhost
+      /               37-56 req/sec
+      /products.rss   395-403 req/sec
+    example.com
+      /               35-58 req/sec
+      /products.rss   400-407 req/sec
+
+View the saved report again with:
+
+  % bong baseline -r log/httperf-report.yml
+
+Lather, rinse, repeat.
+
+== LIMITATIONS
+
+* Can't access pages that require login.
+* HTTP GET only.
+
+== REQUIREMENTS
+
+The httperf command-line tool must be installed. Get it here: 
+
+  http://www.hpl.hp.com/research/linux/httperf/download.php
+
+You must start a webserver (or just Mongrel). Ideally, this would be on a different machine over a fast network. You can also run bong against the local machine or (less ideally) against a remote machine over a slow network, but these will give different performance numbers and may not be accurate.
+
+Internally, bong will
+
+* Run a short series of 5 hits against a URL.
+* Calculate the number of hits needed to run for 10 seconds, or 2 samples. You can change this in the config file, but 2 is the minimum for getting a meaningful standard deviation out of the report.
+* A test will be run again.
+* A short report will be displayed and the raw data will be saved for later comparison.
+
+See http://peepcode.com/products/benchmarking-with-httperf for a full tutorial on httperf (produced by Geoffrey Grosenbach, technical editing by Zed Shaw).
+
+== INSTALL:
+
+* sudo gem install bong
+
+== EXPERIMENTAL GRAPHS
+
+After running bong a number of times, provide a visual output with time on the x axis and req/second on the y axis. Intended to show the change in performance throughout the development of a project.
+
+Assumptions
+
+* Each run will be named "benchmark-1216122887" where the second part can be converted to a Time class using Time.at(..)
+* Runs will be displayed at equal intervals along the x axis regardless of if the time between them is uniform
+* Several URLs may be incldued. rps for each one will be of a similar order of magnitude, so it makes sense to graph them together
+* Not all URLs will have data in all runs. However once a url is added it will be in ALL subsequent runs
+
+== LICENSE:
+
+(The MIT License)
+
+Copyright (c) 2007 Topfunky Corporation
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+'Software'), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/Rakefile

@@ -0,0 +1,16 @@
+require 'rubygems'
+
+require 'hoe'
+require './lib/bong.rb'
+
+Hoe.new('bong', Bong::VERSION) do |p|
+  p.rubyforge_name = 'bong'
+  p.author = 'Geoffrey Grosenbach'
+  p.email = 'boss@topfunky.com'
+  p.summary = 'Website benchmarking helper.'
+  p.description = p.paragraphs_of('README.txt', 1..2).join("\n\n")
+  p.url = "http://rubyforge.org/projects/bong"
+  p.changes = p.paragraphs_of('History.txt', 0..1).join("\n\n")
+
+  p.remote_rdoc_dir = '' # Release to root
+end

data/bin/bong

@@ -0,0 +1,102 @@
+#!/usr/bin/env ruby
+
+##
+# Created on 2007-11-6.
+# Copyright (c) 2007 Topfunky Corporation
+#
+# Permission is hereby granted, free of charge, to any person obtaining
+# a copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish,
+# distribute, sublicense, and/or sell copies of the Software, and to
+# permit persons to whom the Software is furnished to do so, subject to
+# the following conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+begin
+  require "rubygems"
+  require File.dirname(__FILE__) + "/../lib/bong"
+rescue LoadError
+  require File.dirname(__FILE__) + "/../lib/bong"
+end
+
+require 'optparse'
+
+OPTIONS = {
+  :config   => 'config/httperf.yml',
+  :generate => 'config/httperf.yml',
+  :out      => "log/httperf-report.yml"
+}
+MANDATORY_OPTIONS = %w(  )
+
+parser = OptionParser.new do |opts|
+  opts.banner = <<-BANNER
+  A benchmarking tool for staging hits on your website. Uses httperf.
+
+  Usage: #{File.basename($0)} [label] [options]
+
+  label is a name for this benchmarking run.
+    Examples: 'baseline', 'with-optimizations', 'custom-sql-queries'
+
+  Other options are:
+  BANNER
+
+  opts.separator ""
+
+  opts.on("-c", "--config [PATH]", String, "Path to config file.", "Default: config/httperf.yml") do |v|
+    OPTIONS[:config] = v || OPTIONS[:config]
+  end
+
+  opts.on("-g", "--generate [PATH]", String, "Generate a config file.", "Default: config/httperf.yml") do |v|
+    Bong.generate(v || OPTIONS[:generate]); exit
+  end
+
+  opts.on("-o", "--out [PATH]", String, "Write output to a file.") do |v|
+    OPTIONS[:out] = v || OPTIONS[:out]
+  end
+
+  opts.on("-r", "--report PATH", String, "Display the report for a data file.") do |v|
+    OPTIONS[:report] = v
+  end
+
+  opts.on("-t", "--time-report PATH", String, "Create a time graph showing performance over time", "Only works when run in combination with -r") do |v|
+    OPTIONS[:graph] = v
+  end
+  
+  opts.on("-h", "--help", "Show this help message.") { puts opts; exit }
+
+  opts.parse!(ARGV)
+
+  if MANDATORY_OPTIONS && MANDATORY_OPTIONS.find { |option| OPTIONS[option.to_sym].nil? }
+    puts opts; exit
+  end
+
+  OPTIONS[:label] = ARGV.first || "benchmark-#{Time.now.to_i}"
+
+end
+
+# Finally, run the app.
+
+bong = Bong.new(OPTIONS[:config], OPTIONS[:label])
+
+if OPTIONS[:graph] && OPTIONS[:report]
+  bong.graph_report(OPTIONS[:graph], OPTIONS[:report])
+  exit
+elsif OPTIONS[:report]
+  bong.load_report(OPTIONS[:report])
+else
+  bong.run
+  bong.save_report(OPTIONS[:out])
+end
+
+puts bong.report

data/lib/bong.rb

@@ -0,0 +1,225 @@
+require 'yaml'
+require 'logger'
+
+##
+# A tool for running httperf against a website. Documentation coming soon.
+
+class Bong
+  VERSION = '0.0.2'
+
+  ##
+  # Generate a sample config file.
+
+  def self.generate(config_yml_path)
+    config_data = {
+      'servers' => ['localhost:3000'],
+      'uris'    => ['/', '/pages/about'],
+      'samples' => 2
+    }
+
+    if File.exist?(config_yml_path)
+      puts("A config file already exists at '#{config_yml_path}'.")
+      exit
+    end
+
+    File.open(config_yml_path, 'w') do |f|
+      f.write config_data.to_yaml
+    end
+  end
+
+  def initialize(config_yml_path, label)
+    unless File.exist?(config_yml_path)
+      puts <<-MESSAGE
+
+      A config file could not be found at '#{config_yml_path}'.
+
+      Please generate one with the -g option.
+
+      MESSAGE
+      exit
+    end
+
+    @config       = YAML.load(File.read(config_yml_path))
+    @label        = label
+    @stats        = {}
+
+    @logger       = Logger.new(STDOUT)
+    @logger.level = Logger::DEBUG
+    
+    @logger.info "Running suite for label '#{@label}'"
+  end
+
+  def run
+    servers.each do |server, port|
+      port ||= 80
+      @stats[server] = {}
+      uris.each do |uri|
+        run_benchmark(server, port, uri)
+      end
+    end
+  end
+
+
+  def graph_report(graph_path, report_yml_path)
+    # Require gruff here so people can run the rest of the app without gruff.
+    require 'gruff'
+    
+    @report = YAML.load(File.read(report_yml_path))
+
+    # Remove any with no date
+    @report.reject! { |name, data| name.split("-").size != 2 }
+
+    number_of_times = @report.size
+    
+    inverted_report = { }
+    
+    @report.each do |name, data|
+      report_time = Time.at(name.split("-").last.to_i)
+      date_time = report_time.strftime("%d/%m %H:%M")
+      
+      data.each do |host, urls|
+        urls.each do |url, payload|
+          inverted_report[url] ||= { }
+          inverted_report[url][date_time] = payload['avg'] || nil
+        end
+      end      
+    end
+    
+    inverted_report.each do |url, payload|
+      inverted_report[url][:array] = inverted_report[url].to_a.sort.map{|ele| ele.last}
+      missing_times = number_of_times - inverted_report[url][:array].size
+      inverted_report[url][:array] = Array.new(missing_times) + inverted_report[url][:array]
+    end
+      
+    g = Gruff::Line.new
+    g.title = "Requests per second" 
+
+    inverted_report.each do |url, payload|
+      g.data(url, inverted_report[url][:array])
+    end
+    
+    g.write(graph_path)
+  end
+  
+  def load_report(report_yml_path, label=nil)
+    @report = YAML.load(File.read(report_yml_path))
+    label = label || @label || @report.keys.first
+    @stats = @report[label]
+  end
+
+  def report
+    length_of_longest_uri = uris.inject(0) { |length, uri| uri_length = uri.length; (uri_length > length ? uri_length : length) }
+    
+    output = ["\n#{@label}"]
+    servers.each do |server, port|
+      output << "  #{server}"
+      uris.each do |uri|
+        output << "    #{format_string(uri, length_of_longest_uri)} #{format_rounded(@stats[server][uri]['avg_low'])}-#{format_rounded(@stats[server][uri]['avg_high'])} req/sec"
+      end
+    end
+    output.join("\n")
+  end
+
+  def save_report(path)
+    @all_stats = {}
+    if File.exist?(path)
+      @all_stats = YAML.load(File.read(path))
+    end
+    
+    @all_stats[@label] = @stats
+    
+    File.open(path, 'w') do |f|
+      f.write @all_stats.to_yaml
+    end
+  end
+
+  protected
+
+  ##
+  # A list of servers and ports from the config file.
+
+  def servers
+    @config['servers'].map { |server| server.split(/:/) }
+  end
+
+  def uris
+    @config['uris']
+  end
+
+  def run_benchmark(server, port, uri)
+    until (sufficient_sample_size?(server, uri) && no_errors?)
+      increase_num_conns(server, uri)
+      exec_command(server, port, uri, @num_conns)
+      @stats[server][uri] = parse_results
+    end
+  end
+
+  def exec_command(server, port, uri, num_conns)
+    @logger.info "Sending #{@num_conns} hits to #{server}:#{port}#{uri}"
+    cmd = "httperf --server #{server} --port #{port} --uri #{uri} --num-conns #{num_conns}"
+    @output = `#{cmd}`
+  end
+
+  def parse_results
+    stat = {}
+
+    # Total: connections 5 requests 5 replies 5 test-duration 0.013 s
+    stat['duration'] = @output.scan(/test-duration ([\d.]+)/).flatten.first.to_f
+
+    # Reply rate [replies/s]: min 0.0 avg 0.0 max 0.0 stddev 0.0 (0 samples)
+    (stat['min'], stat['avg'], stat['max'], stat['stddev'], stat['samples']) = @output.scan(/Reply rate \[replies\/s\]: min ([\d.]+) avg ([\d.]+) max ([\d.]+) stddev ([\d.]+) \((\d+) samples\)/).flatten.map { |i| i.to_f }
+
+    # Reply status: 1xx=0 2xx=5 3xx=0 4xx=0 5xx=0
+    (stat['1xx'], stat['2xx'], stat['3xx'], stat['4xx'], stat['5xx']) = @output.scan(/Reply status: 1xx=(\d+) 2xx=(\d+) 3xx=(\d+) 4xx=(\d+) 5xx=(\d+)/).flatten.map { |i| i.to_f }
+
+    stat['avg_low']  = stat['avg'].to_f - 2.0 * stat['stddev'].to_f
+    stat['avg_high'] = stat['avg'].to_f + 2.0 * stat['stddev'].to_f
+
+    stat
+  end
+
+  def sufficient_sample_size?(server, uri)
+    @stats[server][uri]['samples'] >= @config['samples'].to_f
+  rescue
+    false
+  end
+
+  def no_errors?
+    # TODO
+    true
+  end
+
+  def increase_num_conns(server, uri)
+    samples             = @stats[server][uri]['samples']
+    duration            = @stats[server][uri]['duration']
+    target_samples      = @config['samples']
+
+    seconds_per_request = (duration / @num_conns.to_f) # 0.02
+    adjusted_conns      = (target_samples * 5.0) / seconds_per_request # 500
+
+    # Increase the connections by the factor and a bit more.
+    @num_conns          = (adjusted_conns * 1.2).to_i
+  rescue
+    @num_conns = 5
+    nil
+  end
+
+  ##
+  # Return a string with rounding for display.
+  #
+  # Small numbers will have a decimal point. Larger numbers will be shown
+  # as plain integers.
+
+  def format_rounded(number)
+    if number > 20
+      number.to_i
+    else
+      sprintf('%0.1f', number)
+    end
+  end
+
+  def format_string(string, length)
+    sprintf "%-#{length + 2}s", string
+  end
+
+end

metadata

@@ -0,0 +1,69 @@
+--- !ruby/object:Gem::Specification 
+name: topfunky-bong
+version: !ruby/object:Gem::Version 
+  version: 0.0.2
+platform: ruby
+authors: 
+- Geoffrey Grosenbach
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2008-07-15 00:00:00 -07:00
+default_executable: bong
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: hoe
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: 1.5.1
+    version: 
+description: Hit your website with bong. Uses httperf to run a suite of benchmarking tests against specified urls on your site.   Graphical output and multi-test comparisons are planned. Apache ab support may be added in the future.
+email: boss@topfunky.com
+executables: 
+- bong
+extensions: []
+
+extra_rdoc_files: 
+- History.txt
+- Manifest.txt
+- README.txt
+files: 
+- History.txt
+- Manifest.txt
+- README.txt
+- Rakefile
+- bin/bong
+- lib/bong.rb
+has_rdoc: true
+homepage: http://rubyforge.org/projects/bong
+post_install_message: 
+rdoc_options: 
+- --main
+- README.txt
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+requirements: []
+
+rubyforge_project: bong
+rubygems_version: 1.2.0
+signing_key: 
+specification_version: 2
+summary: Website benchmarking helper.
+test_files: []
+