request-log-analyzer 1.0.2 → 1.6.3

data/.gitignore

@@ -0,0 +1,10 @@
+.svn/
+.DS_Store
+request-log-analyzer-*.gem
+requests.db
+/pkg
+/doc
+/tmp
+/classes
+/files
+/coverage

data/DESIGN.rdoc

@@ -0,0 +1,41 @@
+=== Request-log-analyzer
+RLA is set up like a simple pipe and filter system.
+
+This allows you to easily add extra reports, filters and outputs.
+                             -> Aggregator  (database)
+  Source -> Filter -> Filter -> Aggregator  (summary report)  -> Output
+                             -> Aggregator  (...)
+
+When the pipeline has been constructed, we Start chunk producer (source) and push requests through pipeline.
+
+  Controller.start
+
+=== Source
+RequestLogAnalyzer::Source is an Object that pushes requests into the chain.
+At the moment you can only use the log-parser as a source.
+It accepts files or stdin and can parse then into request objects using a RequestLogAnalyzer::FileFormat definition.
+In the future we want to be able to have a generated request database as source as this will make interactive
+down drilling possible.
+
+=== Filter
+The filters are all subclasses of the RequestLogAnalyzer::Filter class.
+They accept a request object, manipulate or drop it, and then pass the request object on to the next filter
+in the chain.
+At the moment there are three types of filters available: Anonymize, Field and Timespan.
+
+=== Aggregator
+The Aggregators all inherit from the RequestLogAnalyzer::Aggregator class.
+All the requests that come out of the Filterchain are fed into all the aggregators in parallel.
+These aggregators can do anything what they want with the given request.
+For example: the Database aggregator will just store all the requests into a SQLite database while the Summarizer will
+generate a wide range of statistical reports from them.
+
+=== Running the pipeline
+All Aggregators are asked to report what they have done. For example the database will report: I stuffed x requests
+into SQLite database Y. The Summarizer will output its reports.
+
+  Controller.report
+
+The output is pushed to a RequestLogAnalyzer::Output object, which takes care of the output.
+It can generate either ASCII, UTF8 or even HTML output.
+

data/LICENSE

@@ -1,5 +1,5 @@
-Copyright (c) 2008 Willem van Bergen / Bart ten Brinke
- 
+Copyright (c) 2008-2009 Willem van Bergen & Bart ten Brinke
+
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the
 "Software"), to deal in the Software without restriction, including
@@ -7,10 +7,10 @@ without limitation the rights to use, copy, modify, merge, publish,
 distribute, sublicense, and/or sell copies of the Software, and to
 permit persons to whom the Software is furnished to do so, subject to
 the following conditions:
- 
+
 The above copyright notice and this permission notice shall be
 included in all copies or substantial portions of the Software.
- 
+
 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
 EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
 MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND

data/README.rdoc

@@ -0,0 +1,38 @@
+= Request-log-analyzer
+
+This is a simple command line tool to analyze request log files in various formats to produce a performance report. Its purpose is to find what actions are best candidates for optimization.
+
+* Analyzes log files. Currently supports: Amazon S3, Apache, Delayed::Job, Merb, Mysql, PostgreSQL, Rack, Rails and more.
+* Combines multiple files and decompresses compressed files, which comes in handy if you are using logrotate.
+* Uses several metrics, including cumulative request time, mean request time, process blockers, database and rendering time, HTTP methods and statuses, Rails action cache statistics, etc.) (Sample output: http://wiki.github.com/wvanbergen/request-log-analyzer/sample-output)
+* Low memory footprint and reasonably fast, so it is safe to run on a production server.
+* MIT licensed
+
+See the project wiki at http://wiki.github.com/wvanbergen/request-log-analyzer for documentation and additional information.
+
+== Installation & basic usage
+
+Install request-log-analyzer as a Ruby gem (you might need to run this command
+as root by prepending +sudo+ to it):
+
+  $ gem install request-log-analyzer
+
+To analyze a Rails log file and produce a performance report, run
+request-log-analyzer like this:
+
+  $ request-log-analyzer log/production.log
+
+For more details, other file formats, and available command line options, see the project's wiki at http://wiki.github.com/wvanbergen/request-log-analyzer
+
+== Additional information
+
+Request-log-analyzer was designed and built by Willem van Bergen and Bart ten
+Brinke.
+
+Do you have a rails application that is not performing as it should? If you need
+an expert to analyze your application, feel free to contact either Willem van
+Bergen (willem@railsdoctors.com) or Bart ten Brinke (bart@railsdoctors.com).
+
+* Project wiki at GitHub: http://wiki.github.com/wvanbergen/request-log-analyzer
+* railsdoctors homepage: http://railsdoctors.com
+* wvanbergen's blog posts: http://techblog.floorplanner.com/tag/request-log-analyzer

data/Rakefile

@@ -1,5 +1,8 @@
 Dir[File.dirname(__FILE__) + "/tasks/*.rake"].each { |file| load(file) }
- 
-desc 'Default: run RSpec for request-log-analyzer.'
-task :default => :spec
 
+# Create rake tasks for a gem manages by github. The tasks are created in the
+# gem namespace
+GithubGem::RakeTasks.new(:gem)
+
+# Set the RSpec runner with specdoc output as default task.
+task :default => "spec:specdoc"

data/bin/request-log-analyzer

@@ -1,15 +1,11 @@
-#!/usr/bin/ruby
-require File.dirname(__FILE__) + '/../lib/request_log_analyzer'
-require File.dirname(__FILE__) + '/../lib/cli/command_line_arguments'
-
-def terminal_width(default = 81)
-  IO.popen('stty -a') do |pipe|
-    column_line = pipe.detect { |line| /(\d+) columns/ =~ line }
-    width = column_line ? $1.to_i : default
-  end
-rescue 
-  default  
-end
+#!/usr/bin/env ruby
+# encoding: utf-8
+
+$:.unshift(File.dirname(__FILE__) + '/../lib')
+require 'request_log_analyzer'
+require 'cli/command_line_arguments'
+require 'cli/progressbar'
+require 'cli/tools'
 
 # Parse the arguments given via commandline
 begin
@@ -18,106 +14,108 @@ begin
     command_line.command(:install) do |install|
       install.parameters = 1
     end
-    
+
+    command_line.command(:console) do |cons|
+      cons.option(:database, :alias => :d, :required => true)
+    end
+
     command_line.command(:strip) do |strip|
       strip.minimum_parameters = 1
-      strip.option(:format, :alias => :f, :default => 'rails')      
+      strip.option(:format, :alias => :f, :default => 'rails')
       strip.option(:output, :alias => :o)
       strip.switch(:discard_teaser_lines, :t)
-      strip.switch(:keep_junk_lines, :j)      
-    end
-    
-    command_line.command(:anonymize) do |anonymize|
-      anonymize.minimum_parameters = 1
-      anonymize.option(:format, :alias => :f, :default => 'rails')            
-      anonymize.option(:output, :alias => :o)
-      anonymize.switch(:discard_teaser_lines, :t)
-      anonymize.switch(:keep_junk_lines, :j)      
+      strip.switch(:keep_junk_lines, :j)
     end
-    
-    command_line.option(:format, :alias => :f, :default => 'rails')
+
+    command_line.option(:format, :alias => :f)
+    command_line.option(:apache_format)
+    command_line.option(:rails_format)
+
     command_line.option(:file, :alias => :e)
-    command_line.switch(:assume_correct_order)
-    
+    command_line.option(:mail, :alias => :m)
+    command_line.option(:mailhost, :default => 'localhost')
+    command_line.option(:parse_strategy, :default => 'assume-correct')
+    command_line.option(:yaml)
+    command_line.option(:dump) # To be deprecated
+
     command_line.option(:aggregator, :alias => :a, :multiple => true)
-    command_line.option(:database, :alias => :d)    
+
+    command_line.option(:database, :alias => :d)
+    command_line.switch(:reset_database)
 
     # filtering options
     command_line.option(:select, :multiple => true, :parameters => 2)
     command_line.option(:reject, :multiple => true, :parameters => 2)
     command_line.option(:after)
-    command_line.option(:before)    
-    
+    command_line.option(:before)
+
     command_line.switch(:boring, :b)
-    command_line.option(:report_width, :default => terminal_width - 1)
-    
+    command_line.option(:output, :alias => :o, :default => 'fixedwidth')
+    command_line.option(:report_width,  :default => terminal_width - 1)
+    command_line.option(:report_amount, :default => 20)
+    command_line.option(:report_sort,   :default => 'sum,mean')
+
     command_line.switch(:debug)
-    
-    command_line.minimum_parameters = 1    
+    command_line.switch(:no_progress)
+
+    command_line.minimum_parameters = 1
   end
-  
+
 rescue CommandLine::Error => e
+  puts "Request-log-analyzer, by Willem van Bergen and Bart ten Brinke - version #{RequestLogAnalyzer::VERSION}"
+  puts "Website: http://railsdoctors.com"
+  puts
   puts "ARGUMENT ERROR: " + e.message if e.message
   puts
   puts "Usage: request-log-analyzer [LOGFILES*] <OPTIONS>"
   puts
   puts "Input options:"
-  puts "  --format <format>, -f:     Uses the specified log file format. Defaults to rails."
   puts "  --after <date>             Only consider requests from <date> or later."
-  puts "  --before <date>            Only consider requests before <date>."  
-  puts "  --select <field> <value>   Only consider requests where <field> matches <value>."        
-  puts "  --reject <field> <value>   Only consider requests where <field> does not match <value>."    
+  puts "  --before <date>            Only consider requests before <date>."
+  puts "  --format <format>, -f:     Log file format. amazon_s3, apache, merb, mysql, rack or rails."
+  puts "  --reject <field> <value>   Only consider requests where <field> does not match <value>."
+  puts "  --select <field> <value>   Only consider requests where <field> matches <value>."
   puts
   puts "Output options:"
   puts "  --boring, -b               Output reports without ASCII colors."
   puts "  --database <filename>, -d: Creates an SQLite3 database of all the parsed request information."
   puts "  --debug                    Print debug information while parsing."
-  puts "  --file <filename>          Output to file."
-  puts 
+  puts "  --file <filename>          Redirect output to file."
+  puts "  --mail <emailaddress>      Send report to an email address."
+  puts "  --mailhost <server>        Use the given server as the SMTP server for sending email."
+  puts "  --no-progress              Hide the progress bar."
+  puts "  --output <format>          Output format. Supports 'html' and 'fixed_width'."
+  puts "  --report-width <amount>    Width of ASCII report. Defaults to terminal width."
+  puts "  --report-amount <amount>   Maximum numer of results per report."
+  puts "  --yaml <filename>          Dump the results in YAML format in the given file."
+  puts
   puts "Examples:"
-  puts "  request-log-analyzer development.log"
-  puts "  request-log-analyzer -b mongrel.0.log mongrel.1.log mongrel.2.log "
+  puts "  request-log-analyzer production.log"
+  puts "  request-log-analyzer mongrel.0.log mongrel.1.log --output HTML --mail root@localhost"
   puts "  request-log-analyzer --format merb -d requests.db production.log"
+  puts "  request-log-analyzer mysql_slow_query.log --reject query /SQL_NO_CACHE/"
   puts
   puts "To install rake tasks in your Rails application, "
   puts "run the following command in your application's root directory:"
   puts
   puts "  request-log-analyzer install rails"
-  exit(0) 
-end
-
-def install_rake_tasks(install_type)
-  if install_type == 'rails'
-    require 'ftools'
-    if File.directory?('./lib/tasks/')
-      File.copy(File.dirname(__FILE__) + '/../tasks/request_log_analyzer.rake', './lib/tasks/request_log_analyze.rake')
-      puts "Installed rake tasks."
-      puts "To use, run: rake log:analyze"
-    else
-      puts "Cannot find /lib/tasks folder. Are you in your Rails directory?"
-      puts "Installation aborted."
-    end
-  else
-    raise "Cannot perform this install type! (#{install_type})"
-  end
+  exit(0)
 end
 
-
 case arguments.command
 when :install
   install_rake_tasks(arguments.parameters[0])
+when :console
+  require 'cli/database_console'
+  DatabaseConsole.new(arguments).run!
 when :strip
-  require File.dirname(__FILE__) + '/../lib/request_log_analyzer/log_processor'  
+  require File.dirname(__FILE__) + '/../lib/request_log_analyzer/log_processor'
   RequestLogAnalyzer::LogProcessor.build(:strip, arguments).run!
-when :anonymize
-  require File.dirname(__FILE__) + '/../lib/request_log_analyzer/log_processor'    
-  RequestLogAnalyzer::LogProcessor.build(:anonymize, arguments).run!  
-else 
-  puts "Request log analyzer, by Willem van Bergen and Bart ten Brinke - Version 1.0\n\n"  
-  
-  # Run the request_log_analyzer!
-  RequestLogAnalyzer::Controller.build(arguments, terminal_width).run!
+else
+  puts "Request-log-analyzer, by Willem van Bergen and Bart ten Brinke - version #{RequestLogAnalyzer::VERSION}"
+  puts "Website: http://railsdoctors.com"
+  puts
 
-  puts 
-  puts "Thanks for using request-log-analyzer"
+  # Run the request_log_analyzer!
+  RequestLogAnalyzer::Controller.build_from_arguments(arguments).run!
 end

data/lib/cli/command_line_arguments.rb

@@ -71,7 +71,7 @@ module CommandLine
     def required?
       @required
     end
-    
+
     # Check if flag is optional
     def optional?
       !@required
@@ -85,22 +85,22 @@ module CommandLine
       !@default_value.nil?
     end
   end
-  
+
   class Arguments
 
     class Definition
-      
+
       ENDLESS_PARAMETERS = 99999
-      
+
       attr_reader :commands, :options, :parameters
-      
+
       def initialize(parent)
         @parent = parent
         @options  = {}
         @commands = {}
         @parameters = nil
       end
-      
+
       def [](option_name)
         option_symbol = CommandLine::Option.rewrite(option_name)
         if the_option = @options.detect { |(name, odef)| odef =~ option_symbol }
@@ -109,22 +109,22 @@ module CommandLine
           raise CommandLine::UnknownOption, option_name
         end
       end
-      
+
       def minimum_parameters=(count_specifier)
         @parameters = count_specifier..ENDLESS_PARAMETERS
       end
-            
+
       def parameters=(count_specifier)
         @parameters = count_specifier
       end
-      
-      alias :files= :parameters= 
-      
+
+      alias :files= :parameters=
+
       def option(name, options = {})
         clo = CommandLine::Option.new(name, options)
         @options[clo.name] = clo
       end
-      
+
       def switch(name, switch_alias = nil)
         option(name, :alias => switch_alias, :parameters => 0)
       end
@@ -134,35 +134,35 @@ module CommandLine
         yield(command_definition) if block_given?
         @commands[CommandLine::Option.rewrite(name)] = command_definition
       end
-      
+
       def has_command?(command)
         @commands[CommandLine::Option.rewrite(command)]
       end
     end
-    
-    OPTION_REGEXP  = /^\-\-([A-z0-9-]+)$/;
-    ALIASES_REGEXP = /^\-([A-z0-9]+)$/
-    
+
+    OPTION_REGEXP  = /^\-\-([A-Za-z0-9-]+)$/;
+    ALIASES_REGEXP = /^\-([A-Aa-z0-9]+)$/
+
     attr_reader :definition
     attr_reader :tokens
     attr_reader :command, :options, :parameters
-    
+
     def self.parse(tokens = $*, &block)
       cla = Arguments.new
       cla.define(&block)
-      return cla.parse!(tokens)   
+      return cla.parse!(tokens)
     end
-    
+
     def initialize
       @tokens = []
-      @definition = Definition.new(self)    
-      @current_definition = @definition        
+      @definition = Definition.new(self)
+      @current_definition = @definition
     end
-    
+
     def define(&block)
       yield(@definition)
     end
-    
+
     def [](option)
       if the_option = @options.detect { |(key, value)| key =~ option }
         the_option[1]
@@ -170,31 +170,31 @@ module CommandLine
         @current_definition[option].default_value
       end
     end
-        
+
     def next_token
       @current_token = @tokens.shift
       return @current_token
     end
-    
+
     def next_parameter
       parameter_candidate = @tokens.first
       parameter = (parameter_candidate.nil? || OPTION_REGEXP =~ parameter_candidate || ALIASES_REGEXP =~ parameter_candidate) ? nil : @tokens.shift
       return parameter
     end
-    
+
     def parse!(tokens)
-      @current_definition = @definition         
+      @current_definition = @definition
       @first_token = true
       @tokens      = tokens.clone
-      
+
       @options = {}
       @parameters   = []
       @command = nil
-      
+
       prepare_result!
-      
+
       while next_token
-        
+
         if @first_token && command_definition = @definition.has_command?(@current_token)
           @current_definition = command_definition
           @command = CommandLine::Option.rewrite(@current_token)
@@ -204,34 +204,34 @@ module CommandLine
             when OPTION_REGEXP;  handle_option($1)
             else;                handle_other_parameter(@current_token)
           end
-          @first_token = false          
+          @first_token = false
         end
-          
+
       end
-      
+
       validate_arguments!
-      
-      return self     
+
+      return self
     end
-    
+
     protected
-    
+
     def prepare_result!
       multiple_options = Hash[*@current_definition.options.select { |name, o| o.multiple? }.flatten]
       multiple_options.each { |name, definition| @options[definition] = [] }
     end
-    
+
     def validate_arguments!
       if @current_definition.parameters && !(@current_definition.parameters === @parameters.length)
         raise CommandLine::ParametersOutOfRange.new(@current_definition.parameters, @parameters.length)
       end
-      
+
       required_options = Hash[*@current_definition.options.select { |name, o| o.required? }.flatten]
       required_options.each do |name, definition|
-        raise CommandLine::RequiredOptionMissing, definition unless self[name] 
+        raise CommandLine::RequiredOptionMissing, definition unless self[name]
       end
     end
-    
+
     def handle_alias_expansion(aliases)
       aliases.reverse.scan(/./) do |alias_char|
         if option_definition = @current_definition[alias_char]
@@ -241,24 +241,24 @@ module CommandLine
         end
       end
     end
-    
+
     def handle_other_parameter(parameter)
       @parameters << parameter
     end
-    
+
     def handle_option(option_name)
       option_definition = @current_definition[option_name]
       raise CommandLine::UnknownOption, option_name if option_definition.nil?
-      
+
       if option_definition.multiple?
-        @options[option_definition] << option_definition.parse(self)        
+        @options[option_definition] << option_definition.parse(self)
       else
         @options[option_definition] = option_definition.parse(self)
       end
     end
-        
+
   end
-  
+
   # Commandline parsing errors and exceptions
   class Error < Exception
   end
@@ -267,7 +267,7 @@ module CommandLine
   class RequiredOptionMissing < CommandLine::Error
     def initialize(option)
       super("You have to provide the #{option.name} option!")
-    end    
+    end
   end
 
   # Missing a required file
@@ -282,14 +282,14 @@ module CommandLine
       else
         super("The command expected #{expected} parameters, but found #{actual}!")
       end
-    end    
+    end
   end
 
   # Missing a required flag argument
   class ParameterExpected < CommandLine::Error
     def initialize(option)
       super("The option #{option.inspect} expects a parameter!")
-    end    
+    end
   end
 
   # Encountered an unkown flag
@@ -297,5 +297,5 @@ module CommandLine
     def initialize(option_identifier)
       super("#{option_identifier.inspect} not recognized as a valid option!")
     end
-  end  
+  end
 end

data/lib/cli/database_console.rb

@@ -0,0 +1,26 @@
+
+class DatabaseConsole
+
+  IRB = RUBY_PLATFORM =~ /(:?mswin|mingw)/ ? 'irb.bat' : 'irb'
+
+  def initialize(arguments)
+    @arguments = arguments
+  end
+
+  def run!
+    libraries = ['irb/completion', 'rubygems', './lib/request_log_analyzer', './lib/cli/database_console_init']
+    libaries_string = libraries.map { |l| "-r #{l}" }.join(' ')
+
+    ENV['RLA_DBCONSOLE_DATABASE'] = @arguments[:database]
+    if @arguments[:apache_format]
+      ENV['RLA_DBCONSOLE_FORMAT'] = 'apache'
+      ENV['RLA_DBCONSOLE_FORMAT_ARGUMENT'] = @arguments[:apache_format]
+    else
+      ENV['RLA_DBCONSOLE_FORMAT'] = @arguments[:format]
+    end
+    # ENV['RLA_DBCONSOLE_FORMAT_ARGS'] = arguments['database']
+
+    exec("#{IRB} #{libaries_string} --simple-prompt")
+  end
+end
+

data/lib/cli/database_console_init.rb

@@ -0,0 +1,43 @@
+# Setup the include path
+$:.unshift(File.dirname(__FILE__) + '/..')
+
+$database = RequestLogAnalyzer::Database.new(ENV['RLA_DBCONSOLE_DATABASE'])
+$database.load_database_schema!
+$database.register_default_orm_classes!
+
+require 'cli/tools'
+
+def wordwrap(string, max = 80, indent = "")
+  strings = [""]
+  string.split(", ").each do |item|
+    if strings.last.length == 0 || strings.last.length + item.length <= max
+      strings.last << item << ', '
+    else
+      strings << (item + ', ')
+    end
+  end
+  strings.map(&:strip).join("\n#{indent}").slice(0..-2)
+end
+
+class Request
+  def inspect
+    request_inspect = "Request[id: #{id}]"
+    request_inspect << " <#{lines.first.source.filename}>" if lines.first.source
+
+    inspected_lines = lines.map do |line|
+      inspect_line = "   - #{line.line_type} (line #{line.lineno})"
+      if (inspect_attributes = line.attributes.reject { |(k, v)| [:id, :source_id, :request_id, :lineno].include?(k.to_sym) }).any?
+        inspect_attributes = inspect_attributes.map { |(k,v)| "#{k} = #{v.inspect}" }.join(', ')
+        inspect_line << "\n      " + wordwrap(inspect_attributes, terminal_width - 6, "      ")
+      end
+      inspect_line
+    end
+
+    request_inspect << "\n" << inspected_lines.join("\n") << "\n\n"
+  end
+end
+
+puts "request-log-analyzer database console"
+puts "-------------------------------------"
+puts "The following ActiveRecord classes are available:"
+puts $database.orm_classes.map { |k| k.name.split('::').last }.join(", ")