request-log-analyzer 1.4.0 → 1.4.0.1

data/LICENSE

@@ -1,4 +1,4 @@
-Copyright (c) 2008 Willem van Bergen / Bart ten Brinke
+Copyright (c) 2008-2009 Willem van Bergen & Bart ten Brinke
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

data/README.rdoc

@@ -1,42 +1,37 @@
 = Request-log-analyzer
 
-This is a simple command line tool to analyze request log files of both Rails and
-Merb to produce a performance report. Its purpose is to find what actions are best candidates for optimization.
+This is a simple command line tool to analyze request log files in various formats to produce a performance report. Its purpose is to find what actions are best candidates for optimization.
 
-* Analyzes Rails log files (all versions), Merb logs, or any other log format you specify
-* Combines multiple files and decompresses compressed files (handy if you are using logrotate)
-* Uses several metrics, including cumulative request time, average request time, process blockers, database and rendering time, HTTP methods and statuses, Rails action cache statistics, etc.) (Sample output: http://wiki.github.com/wvanbergen/request-log-analyzer/sample-output)
-* Low memory footprint and reasonably fast (server-safe)
+* Analyzes Rails request logs, Merb request logs, Apache access logs and more, or parses any other log format you specify.
+* Combines multiple files and decompresses compressed files, which comes in handy if you are using logrotate.
+* Uses several metrics, including cumulative request time, mean request time, process blockers, database and rendering time, HTTP methods and statuses, Rails action cache statistics, etc.) (Sample output: http://wiki.github.com/wvanbergen/request-log-analyzer/sample-output)
+* Low memory footprint and reasonably fast, so it is safe to run on a production server.
 * MIT licensed
 
-Request log analyzer was designed and built by Willem van Bergen and Bart ten Brinke.
+See the project wiki at http://wiki.github.com/wvanbergen/request-log-analyzer for documentation and additional information.
 
-== Installation
+== Installation & basic usage
 
-Install request-log-analyzer as a Ruby gem:
+Install request-log-analyzer as a Ruby gem (you might need to run this command
+as root by prepending +sudo+ to it):
 
-  $ sudo gem install request-log-analyzer
+  $ gem install request-log-analyzer
 
-Alternatively, use the gem from the GitHub gem server:
-
-  $ sudo gem install wvanbergen-request-log-analyzer --source http://gems.github.com
-
-To get the best results out of request-log-analyzer, make sure to
-set up logging correctly: http://wiki.github.com/wvanbergen/request-log-analyzer/configure-logging
-for your application.
-
-== Usage
-
-To analyze a log file and produce a performance report, run request-log-analyzer like this:
+To analyze a Rails log file and produce a performance report, run
+request-log-analyzer like this:
 
   $ request-log-analyzer log/production.log
 
-For more details and available command line options, see the project's wiki: http://wiki.github.com/wvanbergen/request-log-analyzer/basic-usage
+For more details, other file formats, and available command line options, see the project's wiki at http://wiki.github.com/wvanbergen/request-log-analyzer
 
 == Additional information
 
-Do you have a rails application that is not performing as it should?
-If you need an expert to analyze your application, feel free to contact either Willem van Bergen (willem@railsdoctors.com) or Bart ten Brinke (bart@railsdoctors.com).
+Request-log-analyzer was designed and built by Willem van Bergen and Bart ten
+Brinke.
+
+Do you have a rails application that is not performing as it should? If you need
+an expert to analyze your application, feel free to contact either Willem van
+Bergen (willem@railsdoctors.com) or Bart ten Brinke (bart@railsdoctors.com).
 
 * Project wiki at GitHub: http://wiki.github.com/wvanbergen/request-log-analyzer
 * railsdoctors homepage: http://railsdoctors.com

data/lib/request_log_analyzer.rb

@@ -11,7 +11,7 @@ module RequestLogAnalyzer
 
   # The current version of request-log-analyzer.
   # Do not change the value by hand; it will be updated automatically by the gem release script.
-  VERSION = "1.4.0"
+  VERSION = "1.4.0.1"
 
   # Loads constants in the RequestLogAnalyzer namespace using self.load_default_class_file(base, const)
   # <tt>const</tt>:: The constant that is not yet loaded in the RequestLogAnalyzer namespace. This should be passed as a string or symbol.

data/lib/request_log_analyzer/aggregator/summarizer.rb

@@ -90,7 +90,7 @@ module RequestLogAnalyzer::Aggregator
     # Call finalize on all trackers. Saves a YAML dump if this is set in  the options.
     def finalize
       @trackers.each { |tracker| tracker.finalize }
-      save_results_dump(options[:dump]) if options[:dump]
+      save_results_dump(options[:yaml]) if options[:yaml]
     end
 
     # Saves the results of all the trackers in YAML format to a file.

data/lib/request_log_analyzer/controller.rb

@@ -30,7 +30,7 @@ module RequestLogAnalyzer
       options[:database]       = arguments[:database]
       options[:reset_database] = arguments[:reset_database]
       options[:debug]          = arguments[:debug]
-      options[:dump]           = arguments[:dump]
+      options[:yaml]           = arguments[:dump]
       options[:parse_strategy] = arguments[:parse_strategy]
       options[:no_progress]    = arguments[:no_progress]
       options[:format]         = arguments[:format]
@@ -80,9 +80,9 @@ module RequestLogAnalyzer
     # * <tt>:database</tt> Database file
     # * <tt>:reset_database</tt> 
     # * <tt>:debug</tt> Enables echo aggregator.
-    # * <tt>:dump</tt> 
+    # * <tt>:yaml</tt> Output to YAML 
     # * <tt>:parse_strategy</tt> 
-    # * <tt>:no_progress</tt> 
+    # * <tt>:no_progress</tt> Do not display the progress bar
     # * <tt>:output</tt> :fixed_width, :html or Output class. Defaults to fixed width.
     # * <tt>:file</tt> Filestring or File or StringIO
     # * <tt>:format</tt> :rails, {:apache => 'FORMATSTRING'}, :merb, etcetera or Format Class. Defaults to :rails.
@@ -100,7 +100,7 @@ module RequestLogAnalyzer
     #   Refactor :database => options[:database], :dump => options[:dump] away from contoller intialization.
     def self.build(options)
       # Defaults
-      options[:output]        ||= :fixed_width
+      options[:output]        ||= 'fixed_width'
       options[:format]        ||= :rails
       options[:aggregator]    ||= [:summarizer]
       options[:report_width]  ||= 80
@@ -108,6 +108,12 @@ module RequestLogAnalyzer
       options[:report_sort]   ||= 'sum,mean'
       options[:boring]        ||= false
       
+      # Backwards compatibility
+      if options[:dump] && options[:yaml].blank?
+        warn "[DEPRECATION] `:dump` is deprecated.  Please use `:yaml` instead."
+        options[:yaml]          = options[:dump]
+      end
+      
       # Set the output class
       output_args   = {}
       output_object = nil
@@ -145,8 +151,9 @@ module RequestLogAnalyzer
       controller = Controller.new(  RequestLogAnalyzer::Source::LogParser.new(file_format, :source_files => options[:source_files]),
                                     { :output => output_instance,
                                       :database => options[:database],                # FUGLY!
-                                      :dump => options[:dump], 
-                                      :reset_database => options[:reset_database]})
+                                      :yaml => options[:yaml], 
+                                      :reset_database => options[:reset_database],
+                                      :no_progress => options[:no_progress]})
 
       # register filters
       if options[:after] || options[:before]
@@ -188,7 +195,7 @@ module RequestLogAnalyzer
     # <tt>format</tt> Logfile format. Defaults to :rails
     # Options are passd on to the LogParser.
     # * <tt>:database</tt> Database the controller should use.
-    # * <tt>:dump</tt> Yaml Dump the contrller should use.
+    # * <tt>:yaml</tt> Yaml Dump the contrller should use.
     # * <tt>:output</tt> All report outputs get << through this output.
     # * <tt>:no_progress</tt> No progress bar
     def initialize(source, options = {})

data/request-log-analyzer.gemspec

@@ -2,8 +2,8 @@ Gem::Specification.new do |s|
   s.name    = "request-log-analyzer"
   
   # Do not set the version and date field manually, this is done by the release script
-  s.version = "1.4.0"
-  s.date    = "2009-09-30"
+  s.version = "1.4.0.1"
+  s.date    = "2009-10-03"
 
   s.rubyforge_project = 'r-l-a'
 
@@ -35,6 +35,6 @@ Gem::Specification.new do |s|
 
   # The files and test_files directives are set automatically by the release script.
   # Do not change them by hand, but make sure to add the files to the git repository.
-  s.files      = %w(spec/unit/filter/anonymize_filter_spec.rb spec/fixtures/rails_22_cached.log lib/request_log_analyzer/line_definition.rb lib/request_log_analyzer/output/html.rb lib/request_log_analyzer/controller.rb spec/lib/macros.rb lib/request_log_analyzer/file_format/rails_development.rb spec/fixtures/apache_combined.log spec/fixtures/apache_common.log spec/fixtures/merb_prefixed.log lib/request_log_analyzer/file_format/amazon_s3.rb tasks/request_log_analyzer.rake spec/unit/file_format/file_format_api_spec.rb spec/unit/file_format/apache_format_spec.rb spec/integration/command_line_usage_spec.rb lib/request_log_analyzer/database.rb spec/fixtures/decompression.log.bz2 spec/fixtures/rails_unordered.log lib/request_log_analyzer/log_processor.rb lib/request_log_analyzer/tracker.rb lib/request_log_analyzer/filter.rb bin/request-log-analyzer request-log-analyzer.gemspec DESIGN.rdoc spec/unit/filter/timespan_filter_spec.rb spec/unit/aggregator/database_inserter_spec.rb spec/lib/matchers.rb lib/request_log_analyzer/filter/field.rb lib/request_log_analyzer/tracker/frequency.rb spec/fixtures/decompression.log.gz spec/fixtures/decompression.log spec/lib/testing_format.rb spec/fixtures/test_order.log spec/fixtures/rails.db lib/request_log_analyzer/output/fixed_width.rb lib/request_log_analyzer/filter/anonymize.rb lib/request_log_analyzer/tracker/timespan.rb lib/request_log_analyzer/database/base.rb lib/request_log_analyzer/aggregator.rb spec/unit/request_spec.rb lib/cli/progressbar.rb lib/request_log_analyzer/mailer.rb README.rdoc lib/request_log_analyzer/database/warning.rb spec/fixtures/merb.log lib/request_log_analyzer/tracker/hourly_spread.rb .gitignore spec/unit/tracker/tracker_api_spec.rb spec/unit/tracker/duration_tracker_spec.rb spec/unit/file_format/amazon_s3_format_spec.rb lib/request_log_analyzer/aggregator/echo.rb spec/unit/mailer_spec.rb spec/unit/controller/log_processor_spec.rb spec/spec_helper.rb lib/request_log_analyzer.rb spec/database.yml Rakefile lib/request_log_analyzer/database/connection.rb spec/unit/filter/filter_spec.rb spec/fixtures/test_language_combined.log lib/request_log_analyzer/aggregator/database_inserter.rb lib/request_log_analyzer/aggregator/summarizer.rb lib/request_log_analyzer/file_format/rack.rb lib/request_log_analyzer/database/source.rb lib/request_log_analyzer/file_format/rails.rb spec/fixtures/decompression.tar.gz spec/unit/tracker/traffic_tracker_spec.rb spec/unit/filter/field_filter_spec.rb spec/unit/database/base_class_spec.rb lib/request_log_analyzer/filter/timespan.rb lib/request_log_analyzer/source/log_parser.rb spec/fixtures/decompression.tgz spec/unit/tracker/timespan_tracker_spec.rb spec/unit/tracker/hourly_spread_spec.rb spec/fixtures/header_and_footer.log lib/cli/tools.rb lib/request_log_analyzer/file_format/merb.rb spec/fixtures/multiple_files_1.log spec/unit/file_format/merb_format_spec.rb spec/unit/file_format/line_definition_spec.rb lib/request_log_analyzer/source.rb lib/request_log_analyzer/request.rb lib/cli/database_console.rb spec/unit/database/connection_spec.rb spec/unit/controller/controller_spec.rb spec/lib/mocks.rb spec/lib/helpers.rb spec/fixtures/rails_1x.log lib/cli/database_console_init.rb lib/request_log_analyzer/output.rb lib/request_log_analyzer/file_format/apache.rb spec/fixtures/decompression.log.zip spec/unit/source/log_parser_spec.rb spec/fixtures/test_file_format.log tasks/github-gem.rake spec/unit/database/database_spec.rb lib/request_log_analyzer/tracker/duration.rb lib/request_log_analyzer/tracker/traffic.rb lib/request_log_analyzer/file_format.rb spec/unit/aggregator/summarizer_spec.rb spec/fixtures/syslog_1x.log spec/fixtures/rails_22.log lib/request_log_analyzer/database/request.rb spec/fixtures/multiple_files_2.log LICENSE lib/request_log_analyzer/source/database_loader.rb spec/unit/tracker/frequency_tracker_spec.rb spec/unit/file_format/rails_format_spec.rb lib/cli/command_line_arguments.rb)
-  s.test_files = %w(spec/unit/filter/anonymize_filter_spec.rb spec/unit/file_format/file_format_api_spec.rb spec/unit/file_format/apache_format_spec.rb spec/integration/command_line_usage_spec.rb spec/unit/filter/timespan_filter_spec.rb spec/unit/aggregator/database_inserter_spec.rb spec/unit/request_spec.rb spec/unit/tracker/tracker_api_spec.rb spec/unit/tracker/duration_tracker_spec.rb spec/unit/file_format/amazon_s3_format_spec.rb spec/unit/mailer_spec.rb spec/unit/controller/log_processor_spec.rb spec/unit/filter/filter_spec.rb spec/unit/tracker/traffic_tracker_spec.rb spec/unit/filter/field_filter_spec.rb spec/unit/database/base_class_spec.rb spec/unit/tracker/timespan_tracker_spec.rb spec/unit/tracker/hourly_spread_spec.rb spec/unit/file_format/merb_format_spec.rb spec/unit/file_format/line_definition_spec.rb spec/unit/database/connection_spec.rb spec/unit/controller/controller_spec.rb spec/unit/source/log_parser_spec.rb spec/unit/database/database_spec.rb spec/unit/aggregator/summarizer_spec.rb spec/unit/tracker/frequency_tracker_spec.rb spec/unit/file_format/rails_format_spec.rb)
+  s.files      = %w(spec/unit/filter/anonymize_filter_spec.rb spec/fixtures/rails_22_cached.log lib/request_log_analyzer/line_definition.rb lib/request_log_analyzer/output/html.rb lib/request_log_analyzer/controller.rb spec/lib/macros.rb lib/request_log_analyzer/file_format/rails_development.rb spec/fixtures/apache_combined.log spec/fixtures/apache_common.log spec/fixtures/merb_prefixed.log lib/request_log_analyzer/file_format/amazon_s3.rb tasks/request_log_analyzer.rake spec/unit/file_format/file_format_api_spec.rb spec/unit/file_format/apache_format_spec.rb spec/integration/command_line_usage_spec.rb lib/request_log_analyzer/database.rb spec/fixtures/decompression.log.bz2 spec/fixtures/rails_unordered.log lib/request_log_analyzer/log_processor.rb lib/request_log_analyzer/tracker.rb lib/request_log_analyzer/filter.rb bin/request-log-analyzer request-log-analyzer.gemspec DESIGN.rdoc spec/unit/filter/timespan_filter_spec.rb spec/unit/aggregator/database_inserter_spec.rb spec/lib/matchers.rb lib/request_log_analyzer/filter/field.rb lib/request_log_analyzer/tracker/frequency.rb spec/fixtures/decompression.log.gz spec/fixtures/decompression.log spec/lib/testing_format.rb spec/fixtures/test_order.log spec/fixtures/rails.db lib/request_log_analyzer/output/fixed_width.rb lib/request_log_analyzer/filter/anonymize.rb lib/request_log_analyzer/tracker/timespan.rb lib/request_log_analyzer/database/base.rb lib/request_log_analyzer/aggregator.rb spec/unit/request_spec.rb lib/cli/progressbar.rb lib/request_log_analyzer/mailer.rb README.rdoc lib/request_log_analyzer/database/warning.rb spec/fixtures/merb.log lib/request_log_analyzer/tracker/hourly_spread.rb .gitignore spec/unit/tracker/tracker_api_spec.rb spec/unit/tracker/duration_tracker_spec.rb spec/unit/file_format/amazon_s3_format_spec.rb spec/integration/scout_spec.rb lib/request_log_analyzer/aggregator/echo.rb spec/unit/mailer_spec.rb spec/unit/controller/log_processor_spec.rb spec/spec_helper.rb lib/request_log_analyzer.rb spec/database.yml Rakefile lib/request_log_analyzer/database/connection.rb spec/unit/filter/filter_spec.rb spec/fixtures/test_language_combined.log lib/request_log_analyzer/aggregator/database_inserter.rb lib/request_log_analyzer/aggregator/summarizer.rb lib/request_log_analyzer/file_format/rack.rb lib/request_log_analyzer/database/source.rb lib/request_log_analyzer/file_format/rails.rb spec/fixtures/decompression.tar.gz spec/unit/tracker/traffic_tracker_spec.rb spec/unit/filter/field_filter_spec.rb spec/unit/database/base_class_spec.rb lib/request_log_analyzer/filter/timespan.rb lib/request_log_analyzer/source/log_parser.rb spec/fixtures/decompression.tgz spec/unit/tracker/timespan_tracker_spec.rb spec/unit/tracker/hourly_spread_spec.rb spec/fixtures/header_and_footer.log lib/cli/tools.rb lib/request_log_analyzer/file_format/merb.rb spec/fixtures/multiple_files_1.log spec/unit/file_format/merb_format_spec.rb spec/unit/file_format/line_definition_spec.rb lib/request_log_analyzer/source.rb lib/request_log_analyzer/request.rb lib/cli/database_console.rb spec/unit/database/connection_spec.rb spec/unit/controller/controller_spec.rb spec/lib/mocks.rb spec/lib/helpers.rb spec/fixtures/rails_1x.log lib/cli/database_console_init.rb lib/request_log_analyzer/output.rb lib/request_log_analyzer/file_format/apache.rb spec/fixtures/decompression.log.zip spec/unit/source/log_parser_spec.rb spec/integration/scouts_custom_output.rb spec/fixtures/test_file_format.log tasks/github-gem.rake spec/unit/database/database_spec.rb spec/integration/munin_plugins_rails_spec.rb lib/request_log_analyzer/tracker/duration.rb lib/request_log_analyzer/tracker/traffic.rb lib/request_log_analyzer/file_format.rb spec/unit/aggregator/summarizer_spec.rb spec/fixtures/syslog_1x.log spec/fixtures/rails_22.log lib/request_log_analyzer/database/request.rb spec/fixtures/multiple_files_2.log LICENSE lib/request_log_analyzer/source/database_loader.rb spec/unit/tracker/frequency_tracker_spec.rb spec/unit/file_format/rails_format_spec.rb lib/cli/command_line_arguments.rb)
+  s.test_files = %w(spec/unit/filter/anonymize_filter_spec.rb spec/unit/file_format/file_format_api_spec.rb spec/unit/file_format/apache_format_spec.rb spec/integration/command_line_usage_spec.rb spec/unit/filter/timespan_filter_spec.rb spec/unit/aggregator/database_inserter_spec.rb spec/unit/request_spec.rb spec/unit/tracker/tracker_api_spec.rb spec/unit/tracker/duration_tracker_spec.rb spec/unit/file_format/amazon_s3_format_spec.rb spec/integration/scout_spec.rb spec/unit/mailer_spec.rb spec/unit/controller/log_processor_spec.rb spec/unit/filter/filter_spec.rb spec/unit/tracker/traffic_tracker_spec.rb spec/unit/filter/field_filter_spec.rb spec/unit/database/base_class_spec.rb spec/unit/tracker/timespan_tracker_spec.rb spec/unit/tracker/hourly_spread_spec.rb spec/unit/file_format/merb_format_spec.rb spec/unit/file_format/line_definition_spec.rb spec/unit/database/connection_spec.rb spec/unit/controller/controller_spec.rb spec/unit/source/log_parser_spec.rb spec/unit/database/database_spec.rb spec/integration/munin_plugins_rails_spec.rb spec/unit/aggregator/summarizer_spec.rb spec/unit/tracker/frequency_tracker_spec.rb spec/unit/file_format/rails_format_spec.rb)
 end

data/spec/integration/command_line_usage_spec.rb

@@ -72,9 +72,9 @@ describe RequestLogAnalyzer, 'running from command line' do
   end
 
   it "should dump the results to a YAML file" do
-    run("#{log_fixture(:rails_1x)} --dump #{temp_output_file(:dump)}")
-    File.exist?(temp_output_file(:dump)).should be_true
-    YAML::load(File.read(temp_output_file(:dump))).should have_at_least(1).item
+    run("#{log_fixture(:rails_1x)} --dump #{temp_output_file(:yaml)}")
+    File.exist?(temp_output_file(:yaml)).should be_true
+    YAML::load(File.read(temp_output_file(:yaml))).should have_at_least(1).item
   end
 
   it "should parse 4 requests from the standard input" do

data/spec/integration/munin_plugins_rails_spec.rb

@@ -0,0 +1,58 @@
+require File.dirname(__FILE__) + '/../spec_helper.rb'
+
+describe RequestLogAnalyzer, 'when harvesting like munin-plugins-rails the YAML output' do
+  
+  before(:each) do 
+    cleanup_temp_files!    
+    run("#{log_fixture(:rails_1x)} --dump #{temp_output_file(:yaml)}")
+    @rla = YAML::load(File.read(temp_output_file(:yaml)))
+  end
+  
+  after(:each) do
+    cleanup_temp_files!
+  end
+  
+  it "should contain database times" do
+    @rla["Database time"].each do |item|
+      item[1][:min].should_not be_nil
+      item[1][:max].should_not be_nil
+      item[1][:hits].should_not be_nil
+      item[1][:sum].should_not be_nil
+    end
+  end
+
+  it "should contain request times" do
+    @rla["Request duration"].each do |item|
+      item[1][:min].should_not be_nil
+      item[1][:max].should_not be_nil
+      item[1][:hits].should_not be_nil
+      item[1][:sum].should_not be_nil
+    end
+  end
+
+  it "should contain failed requests" do
+    @rla.keys.should include("Failed requests")
+  end
+
+  it "should contain Process blockers" do
+    @rla.keys.should include("Process blockers (> 1 sec duration)")
+  end
+
+  it "should contain HTTP Methods" do
+    @rla["HTTP methods"]["GET"].should_not be_nil
+  end
+
+  it "should contain HTTP Methods" do
+    @rla["HTTP methods"]["GET"].should_not be_nil
+  end
+
+  it "should contain view rendering times" do
+    @rla["View rendering time"].each do |item|
+      item[1][:min].should_not be_nil
+      item[1][:max].should_not be_nil
+      item[1][:hits].should_not be_nil
+      item[1][:sum].should_not be_nil
+    end
+  end
+
+end

data/spec/integration/scout_spec.rb

@@ -0,0 +1,71 @@
+require File.dirname(__FILE__) + '/../spec_helper.rb'
+require File.dirname(__FILE__) + '/scouts_custom_output'
+
+def capture_stdout_and_stderr_with_warnings_on
+  $stdout, $stderr, warnings, $VERBOSE =
+    StringIO.new, StringIO.new, $VERBOSE, true
+  yield
+  return $stdout.string, $stderr.string
+ensure
+  $stdout, $stderr, $VERBOSE = STDOUT, STDERR, warnings
+end
+
+describe RequestLogAnalyzer, 'when using the rla API like the scout plugin' do
+  
+  before(:each) do
+    # prepare a place to capture the output
+    sio = StringIO.new
+    
+    # place an IO object where I want RequestLogAnalyzer to read from
+    open(log_fixture(:rails_1x)) do |log|
+      completed_count = 0
+      log.each do |line|
+        completed_count += 1 if line =~ /\ACompleted\b/
+        break if completed_count == 2  # skipping first two requests
+      end
+      
+      # trigger the log parse
+      @stdout, @stderr = capture_stdout_and_stderr_with_warnings_on do
+        RequestLogAnalyzer::Controller.build(
+          :output       => EmbeddedHTML,
+          :file         => sio,
+          :after        => Time.local(2008, 8, 14, 21, 16, 31),  # after 3rd req
+          :source_files => log
+        ).run!
+      end
+    end
+    
+    # read the resulting output
+    @analysis = sio.string
+  end
+  
+  it "should generate an analysis" do
+    @analysis.should_not be_empty
+  end
+  
+  it "should generate customized output using the passed Class" do
+    credit = %r{<p>Powered by request-log-analyzer v\d+(?:\.\d+)+</p>\z}
+    @analysis.should match(credit)
+  end
+  
+  it "should skip requests before :after Time" do
+    @analysis.should_not include("PeopleController#show")
+  end
+  
+  it "should include requests after IO#pos and :after Time" do
+    @analysis.should include("PeopleController#picture")
+  end
+  
+  it "should skip requests before IO#pos" do
+    @analysis.should_not include("PeopleController#index")
+  end
+  
+  it "should not print to $stdout" do
+    @stdout.should be_empty
+  end
+  
+  it "should not print to $stderr (with warnings on)" do
+    @stderr.should be_empty
+  end
+
+end

data/spec/integration/scouts_custom_output.rb

@@ -0,0 +1,78 @@
+class EmbeddedHTML < RequestLogAnalyzer::Output::Base
+  def print(str)
+    @io << str
+  end
+  alias_method :<<, :print
+
+  def puts(str = "")
+    @io << "#{str}<br/>\n"
+  end
+
+  def title(title)
+    @io.puts(tag(:h2, title))
+  end
+
+  def line(*font)  
+    @io.puts(tag(:hr))
+  end
+
+  def link(text, url = nil)
+    url = text if url.nil?
+    tag(:a, text, :href => url)
+  end
+
+  def table(*columns, &block)
+    rows = Array.new
+    yield(rows)
+
+    @io << tag(:table, :cellspacing => 0) do |content|
+      if table_has_header?(columns)
+        content << tag(:tr) do
+          columns.map { |col| tag(:th, col[:title]) }.join("\n")
+        end
+      end
+
+      odd = false
+      rows.each do |row|
+        odd = !odd
+        content << tag(:tr) do
+          if odd
+            row.map { |cell| tag(:td, cell, :class => "alt") }.join("\n") 
+          else
+            row.map { |cell| tag(:td, cell) }.join("\n") 
+          end
+        end
+      end
+    end
+  end
+
+  def header
+  end
+
+  def footer
+    @io << tag(:hr) << tag(:p, "Powered by request-log-analyzer v#{RequestLogAnalyzer::VERSION}")
+  end
+
+  private
+
+  def tag(tag, content = nil, attributes = nil)
+    if block_given?
+      attributes = content.nil? ? "" : " " + content.map { |(key, value)| "#{key}=\"#{value}\"" }.join(" ")
+      content_string = ""
+      content = yield(content_string)
+      content = content_string unless content_string.empty? 
+      "<#{tag}#{attributes}>#{content}</#{tag}>"
+    else
+      attributes = attributes.nil? ? "" : " " + attributes.map { |(key, value)| "#{key}=\"#{value}\"" }.join(" ")
+      if content.nil?
+        "<#{tag}#{attributes} />"
+      else
+        if content.class == Float
+          "<#{tag}#{attributes}><div class='color_bar' style=\"width:#{(content*200).floor}px;\"/></#{tag}>"
+        else
+          "<#{tag}#{attributes}>#{content}</#{tag}>"
+        end
+      end
+    end
+  end  
+end

data/spec/lib/helpers.rb

@@ -10,6 +10,11 @@ module RequestLogAnalyzer::Spec::Helpers
     File.dirname(__FILE__) + "/../fixtures/#{name}.#{extention}"
   end
 
+  # Creates a log file given some lines
+  def log_stream(*lines)
+    StringIO.new(lines.join("\n") + "\n")
+  end
+
   # Request loopback
   def request(fields, format = testing_format)
     if fields.kind_of?(Array)

data/spec/lib/mocks.rb

@@ -70,5 +70,7 @@ module RequestLogAnalyzer::Spec::Mocks
     return connection
   end
 
-
+  def request_counter
+    @request_counter ||= mock('aggregator to count request')
+  end
 end

data/spec/unit/file_format/apache_format_spec.rb

@@ -51,146 +51,153 @@ describe RequestLogAnalyzer::FileFormat::Apache do
 
   context '"Common" access log parsing' do
     before(:all) do
-      @file_format = RequestLogAnalyzer::FileFormat.load(:apache, :common)
-      @log_parser = RequestLogAnalyzer::Source::LogParser.new(@file_format)
-      @sample_1 = '1.129.119.13 - - [08/Sep/2009:07:54:09 -0400] "GET /profile/18543424 HTTP/1.0" 200 8223'
-      @sample_2 = '1.82.235.29 - - [08/Sep/2009:07:54:05 -0400] "GET /gallery/fresh?page=23&per_page=16 HTTP/1.1" 200 23414'
+      @file_format     = RequestLogAnalyzer::FileFormat.load(:apache, :common)
+      @log_parser      = RequestLogAnalyzer::Source::LogParser.new(@file_format)
+      @sample_1        = '1.129.119.13 - - [08/Sep/2009:07:54:09 -0400] "GET /profile/18543424 HTTP/1.0" 200 8223'
+      @sample_2        = '1.82.235.29 - - [08/Sep/2009:07:54:05 -0400] "GET /gallery/fresh?page=23&per_page=16 HTTP/1.1" 200 23414'
+      @nonsense_sample = 'addasdsasadadssadasd'
     end
 
     it "should have a valid language definitions" do
       @file_format.should be_valid
     end
 
-    it "should parse a valid access log line" do
-      @file_format.line_definitions[:access].matches(@sample_1).should be_kind_of(Hash)
-    end
-
     it "should not parse a valid access log line" do
-      @file_format.line_definitions[:access].matches('addasdsasadadssadasd').should be_false
+      @file_format.should_not parse_line(@nonsense_sample)
     end
 
     it "should read the correct values from a valid HTTP/1.0 access log line" do
-      @log_parser.parse_io(StringIO.new(@sample_1)) do |request|
-        request[:remote_host].should  == '1.129.119.13'
-        request[:timestamp].should    == 20090908075409
-        request[:http_status].should  == 200
-        request[:http_method].should  == 'GET'
-        request[:http_version].should == '1.0'
-        request[:bytes_sent].should   == 8223
-        request[:user].should         == nil
-      end
+      @file_format.should parse_line(@sample_1).as(:access).and_capture(
+        :remote_host    => '1.129.119.13',
+        :remote_logname => nil,
+        :user           => nil,
+        :timestamp      => 20090908075409,
+        :http_status    => 200,
+        :http_method    => 'GET',
+        :http_version   => '1.0',
+        :bytes_sent     => 8223)
     end
 
     it "should read the correct values from a valid 200 access log line" do
-      @log_parser.parse_io(StringIO.new(@sample_2)) do |request|
-        request[:remote_host].should  == '1.82.235.29'
-        request[:timestamp].should    == 20090908075405
-        request[:http_status].should  == 200
-        request[:http_method].should  == 'GET'
-        request[:http_version].should == '1.1'
-        request[:bytes_sent].should   == 23414
-        request[:user].should         == nil
+      @file_format.should parse_line(@sample_2).as(:access).and_capture(
+        :remote_host    => '1.82.235.29',
+        :remote_logname => nil,
+        :user           => nil,
+        :timestamp      => 20090908075405,
+        :http_status    => 200,
+        :http_method    => 'GET',
+        :http_version   => '1.1',
+        :bytes_sent     => 23414)
+    end
+
+    it "should parse 10 request from fixture access log without warnings" do
+      request_counter.should_receive(:hit!).exactly(10).times
+      @log_parser.should_not_receive(:warn)
+      
+      @log_parser.parse_file(log_fixture(:apache_common)) do |request| 
+        request_counter.hit! if request.kind_of?(RequestLogAnalyzer::FileFormat::Apache::Request)
       end
     end
-
-    it "should parse 10 request from fixture access log" do
-      counter = mock('counter')
-      counter.should_receive(:hit!).exactly(10).times
-      @log_parser.parse_file(log_fixture(:apache_common)) { counter.hit! }
-    end
   end
 
-  context '"Rack" access log parser' do
-    before(:each) do
-      @file_format = RequestLogAnalyzer::FileFormat.load(:rack)
-      @log_parser  = RequestLogAnalyzer::Source::LogParser.new(@file_format)
-      @sample_1 = '127.0.0.1 - - [16/Sep/2009 06:40:08] "GET /favicon.ico HTTP/1.1" 500 63183 0.0453'
-    end
+  context '"Combined" access log parsing' do
 
-    it "should create a kind of an Apache file format" do
-      @file_format.should be_kind_of(RequestLogAnalyzer::FileFormat::Apache)
+    before(:all) do
+      @file_format = RequestLogAnalyzer::FileFormat.load(:apache, :combined)
+      @log_parser  = RequestLogAnalyzer::Source::LogParser.new(@file_format)
+      @sample_1 = '69.41.0.45 - - [02/Sep/2009:12:02:40 +0200] "GET //phpMyAdmin/ HTTP/1.1" 404 209 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)"'
+      @sample_2 = '10.0.1.1 - - [02/Sep/2009:05:08:33 +0200] "GET / HTTP/1.1" 200 30 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; en-us) AppleWebKit/531.9 (KHTML, like Gecko) Version/4.0.3 Safari/531.9"'
+      @nonsense_sample = 'addasdsasadadssadasd'
     end
 
     it "should have a valid language definitions" do
       @file_format.should be_valid
     end
 
-    it "should parse a valid access log line" do
-      @file_format.line_definitions[:access].matches(@sample_1).should be_kind_of(Hash)
-    end
-
     it "should not parse a valid access log line" do
-      @file_format.line_definitions[:access].matches('addasdsasadadssadasd').should be_false
+      @file_format.should_not parse_line(@nonsense_sample)
     end
 
     it "should read the correct values from a valid 404 access log line" do
-      @log_parser.parse_io(StringIO.new(@sample_1)) do |request|
-        request[:remote_host].should  == '127.0.0.1'
-        request[:timestamp].should    == 20090916064008
-        request[:http_status].should  == 500
-        request[:http_method].should  == 'GET'
-        request[:http_version].should == '1.1'
-        request[:bytes_sent].should   == 63183
-        request[:user].should         == nil
-        request[:duration].should     == 0.0453
+      @file_format.should parse_line(@sample_1).as(:access).and_capture(
+        :remote_host    => '69.41.0.45',
+        :remote_logname => nil,
+        :user           => nil,
+        :timestamp      => 20090902120240,
+        :http_status    => 404,
+        :http_method    => 'GET',
+        :http_version   => '1.1',
+        :bytes_sent     => 209,
+        :referer        => nil,
+        :user_agent     => 'Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)')
+    end
+
+    it "should read the correct values from a valid 200 access log line" do
+      @file_format.should parse_line(@sample_2).as(:access).and_capture(
+        :remote_host    => '10.0.1.1',
+        :remote_logname => nil,
+        :user           => nil,
+        :timestamp      => 20090902050833,
+        :http_status    => 200,
+        :http_method    => 'GET',
+        :http_version   => '1.1',
+        :bytes_sent     => 30,
+        :referer        => nil,
+        :user_agent     => 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; en-us) AppleWebKit/531.9 (KHTML, like Gecko) Version/4.0.3 Safari/531.9')
+    end
+
+    it "should parse 5 request from fixture access log without warnings" do
+      request_counter.should_receive(:hit!).exactly(5).times
+      @log_parser.should_not_receive(:warn)
+      
+      @log_parser.parse_file(log_fixture(:apache_combined)) do |request|
+        request_counter.hit! if request.kind_of?(RequestLogAnalyzer::FileFormat::Apache::Request)
       end
     end
   end
-
-  context '"Combined" access log parsing' do
-
-    before(:all) do
-      @file_format = RequestLogAnalyzer::FileFormat.load(:apache, :combined)
+  
+  context '"Rack" access log parser' do
+    before(:each) do
+      @file_format = RequestLogAnalyzer::FileFormat.load(:rack)
       @log_parser  = RequestLogAnalyzer::Source::LogParser.new(@file_format)
-      @sample_1 = '69.41.0.45 - - [02/Sep/2009:12:02:40 +0200] "GET //phpMyAdmin/ HTTP/1.1" 404 209 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)"'
-      @sample_2 = '10.0.1.1 - - [02/Sep/2009:05:08:33 +0200] "GET / HTTP/1.1" 200 30 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; en-us) AppleWebKit/531.9 (KHTML, like Gecko) Version/4.0.3 Safari/531.9"'
+      @sample_1 = '127.0.0.1 - - [16/Sep/2009 07:40:08] "GET /favicon.ico HTTP/1.1" 500 63183 0.0453'
+      @sample_2 = '127.0.0.1 - - [01/Oct/2009 07:58:10] "GET / HTTP/1.1" 200 1 0.0045'
+      @nonsense_sample = 'addasdsasadadssadasd'
+    end
+
+    it "should create a kind of an Apache file format" do
+      @file_format.should be_kind_of(RequestLogAnalyzer::FileFormat::Apache)
     end
 
     it "should have a valid language definitions" do
       @file_format.should be_valid
     end
 
-    it "should parse a valid access log line" do
-      @file_format.line_definitions[:access].matches(@sample_1).should be_kind_of(Hash)
+    it "should parse a valid access log line with status 500" do
+      @file_format.should parse_line(@sample_1).as(:access).and_capture(
+        :remote_host => '127.0.0.1', :timestamp => 20090916074008, :user => nil,
+        :http_status => 500,         :http_method => 'GET',        :http_version => '1.1',
+        :duration => 0.0453,         :bytes_sent => 63183,         :remote_logname => nil)
     end
 
-    it "should not parse a valid access log line" do
-      @file_format.line_definitions[:access].matches('addasdsasadadssadasd').should be_false
+    it "should parse a valid access log line with status 200" do
+      @file_format.should parse_line(@sample_2).as(:access).and_capture(
+        :remote_host => '127.0.0.1', :timestamp => 20091001075810, :user => nil,
+        :http_status => 200,         :http_method => 'GET',        :http_version => '1.1',
+        :duration => 0.0045,         :bytes_sent => 1,             :remote_logname => nil)
     end
 
-    it "should read the correct values from a valid 404 access log line" do
-      @log_parser.parse_io(StringIO.new(@sample_1)) do |request|
-        request[:remote_host].should  == '69.41.0.45'
-        request[:timestamp].should    == 20090902120240
-        request[:http_status].should  == 404
-        request[:http_method].should  == 'GET'
-        request[:http_version].should == '1.1'
-        request[:bytes_sent].should   == 209
-        request[:referer].should      == nil
-        request[:user].should         == nil
-        request[:user_agent].should   == 'Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)'
-      end
+    it "should not parse an invalid access log line" do
+      @file_format.should_not parse_line(@nonsense_sample)
     end
 
-    it "should read the correct values from a valid 200 access log line" do
-      @log_parser.parse_io(StringIO.new(@sample_2)) do |request|
-        request[:remote_host].should  == '10.0.1.1'
-        request[:timestamp].should    == 20090902050833
-        request[:http_status].should  == 200
-        request[:http_method].should  == 'GET'
-        request[:http_version].should == '1.1'
-        request[:bytes_sent].should   == 30
-        request[:referer].should      == nil
-        request[:user].should         == nil
-        request[:user_agent].should   == 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; en-us) AppleWebKit/531.9 (KHTML, like Gecko) Version/4.0.3 Safari/531.9'
+    it "should parse 2 Apache requests from a sample without warnings" do
+      request_counter.should_receive(:hit!).twice
+      @log_parser.should_not_receive(:warn)
+      
+      @log_parser.parse_io(log_stream(@sample_1, @nonsense_sample, @sample_2)) do |request|
+        request_counter.hit! if request.kind_of?(RequestLogAnalyzer::FileFormat::Apache::Request)
       end
     end
-
-    it "should parse 5 request from fixture access log" do
-      counter = mock('counter')
-      counter.should_receive(:hit!).exactly(5).times
-      @log_parser.parse_file(log_fixture(:apache_combined)) { counter.hit! }
-    end
   end
 end
-

data/spec/unit/file_format/rails_format_spec.rb

@@ -120,35 +120,32 @@ describe RequestLogAnalyzer::FileFormat::Rails do
     end
 
     it "should parse a Rails 2.1 style log and find valid Rails requests without warnings" do
+      request_counter.should_receive(:hit!).exactly(4).times
       @log_parser.should_not_receive(:warn)
+      
       @log_parser.parse_file(log_fixture(:rails_1x)) do |request|
-        request.should be_kind_of(RequestLogAnalyzer::FileFormat::Rails::Request)
-        request.should be_completed
+        request_counter.hit! if request.kind_of?(RequestLogAnalyzer::FileFormat::Rails::Request) && request.completed?
       end
     end
 
     it "should parse a Rails 2.2 style log and find valid Rails requests without warnings" do
+      request_counter.should_receive(:hit!).once
       @log_parser.should_not_receive(:warn)
+      
       @log_parser.parse_file(log_fixture(:rails_22)) do |request|
-        request.should be_kind_of(RequestLogAnalyzer::FileFormat::Rails::Request)
-        request.should be_completed
+        request_counter.hit! if request.kind_of?(RequestLogAnalyzer::FileFormat::Rails::Request) && request.completed?
       end
     end
 
     it "should parse a Rails SyslogLogger file with prefix and find valid requests without warnings" do
+      request_counter.should_receive(:hit!).once
       @log_parser.should_not_receive(:warn)
+      
       @log_parser.parse_file(log_fixture(:syslog_1x)) do |request|
-        request.should be_kind_of(RequestLogAnalyzer::FileFormat::Rails::Request)
-        request.should be_completed
+        request_counter.hit! if request.kind_of?(RequestLogAnalyzer::FileFormat::Rails::Request) && request.completed?
       end
     end
 
-    it "should find 4 completed requests" do
-      @log_parser.should_not_receive(:warn)
-      @log_parser.should_receive(:handle_request).exactly(4).times
-      @log_parser.parse_file(log_fixture(:rails_1x))
-    end
-
     it "should parse cached requests" do
       @log_parser.should_not_receive(:warn)
       @log_parser.parse_file(log_fixture(:rails_22_cached)) do |request|

data/spec/unit/tracker/tracker_api_spec.rb

@@ -116,7 +116,7 @@ describe RequestLogAnalyzer::Tracker::Base do
     end
 
     it "should receive :to_yaml object when finalizing" do
-      @summarizer.options[:dump] = temp_output_file(:dump)
+      @summarizer.options[:yaml] = temp_output_file(:yaml)
       @tracker.should_receive(:to_yaml_object).once
       @summarizer.to_yaml
     end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification 
 name: request-log-analyzer
 version: !ruby/object:Gem::Version 
-  version: 1.4.0
+  version: 1.4.0.1
 platform: ruby
 authors: 
 - Willem van Bergen
@@ -10,7 +10,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2009-09-30 00:00:00 +02:00
+date: 2009-10-03 00:00:00 +02:00
 default_executable: request-log-analyzer
 dependencies: 
 - !ruby/object:Gem::Dependency 
@@ -94,6 +94,7 @@ files:
 - spec/unit/tracker/tracker_api_spec.rb
 - spec/unit/tracker/duration_tracker_spec.rb
 - spec/unit/file_format/amazon_s3_format_spec.rb
+- spec/integration/scout_spec.rb
 - lib/request_log_analyzer/aggregator/echo.rb
 - spec/unit/mailer_spec.rb
 - spec/unit/controller/log_processor_spec.rb
@@ -137,9 +138,11 @@ files:
 - lib/request_log_analyzer/file_format/apache.rb
 - spec/fixtures/decompression.log.zip
 - spec/unit/source/log_parser_spec.rb
+- spec/integration/scouts_custom_output.rb
 - spec/fixtures/test_file_format.log
 - tasks/github-gem.rake
 - spec/unit/database/database_spec.rb
+- spec/integration/munin_plugins_rails_spec.rb
 - lib/request_log_analyzer/tracker/duration.rb
 - lib/request_log_analyzer/tracker/traffic.rb
 - lib/request_log_analyzer/file_format.rb
@@ -197,6 +200,7 @@ test_files:
 - spec/unit/tracker/tracker_api_spec.rb
 - spec/unit/tracker/duration_tracker_spec.rb
 - spec/unit/file_format/amazon_s3_format_spec.rb
+- spec/integration/scout_spec.rb
 - spec/unit/mailer_spec.rb
 - spec/unit/controller/log_processor_spec.rb
 - spec/unit/filter/filter_spec.rb
@@ -211,6 +215,7 @@ test_files:
 - spec/unit/controller/controller_spec.rb
 - spec/unit/source/log_parser_spec.rb
 - spec/unit/database/database_spec.rb
+- spec/integration/munin_plugins_rails_spec.rb
 - spec/unit/aggregator/summarizer_spec.rb
 - spec/unit/tracker/frequency_tracker_spec.rb
 - spec/unit/file_format/rails_format_spec.rb