wvanbergen-request-log-analyzer 0.1.0

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2008 Willem van Bergen / Bart ten Brinke
+ 
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+ 
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+ 
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README

@@ -0,0 +1,119 @@
+Request log analyzer
+--------------------------------
+
+This is a simple command line tool to analyze request log files. At this moment,
+it only supports Rails log files, but Merb log files are planned to be supported
+as well. Its purpose is to find what actions are best candidates for optimization.
+
+This tool will parse all requests in the logfile and aggregate the
+information. Once it is finished parsing the log file, it will show the
+requests that take op most server time. Different metrics are used (cumulative
+time, average time, blockers, DB time, etc)
+
+
+Installation
+--------------------------------
+gem sources -a http://gems.github.com
+sudo gem install wvanbergen-request-log-analyzer
+
+Usage
+--------------------------------
+
+Usage: request-log-analyzer [FILE] [OPTION]
+Analyze the given log FILE with the given OPTION
+Example: request-log-analyzer mongrel.log
+
+  --fast, -t:                 Only use completed requests
+  --guess-database-time, -g:  Guesses the database duration of requests if they are not in the log
+  --output, -o:               Comma-separated list of reports to show    
+  --amount, -c:               Displays the top <amount> elements in the reports
+  --colorize, -z:             Fancy bash coloring
+
+
+
+Example
+--------------------------------
+
+Note that this example was shortened for your viewing pleasure.
+$ request-log-analyzer /var/log/my_app.log
+
+Request log analyzer, by Willem van Bergen and  Bart ten Brinke
+
+Processing all log lines...
+========================================================================
+Successfully analyzed 58908 requests from log file
+
+Timestamp first request: 2008-07-13T06:25:58+00:00
+Timestamp last request:  2008-07-20T06:18:53+00:00
+Total time analyzed: 7 days
+
+Top 10 most requested actions
+========================================================================
+/overview/:date/                                  : 19359 requests
+/overview/day/:date/                              : 6365 requests
+/overview/:date/set/                              : 5589 requests
+/overview/                                        : 3985 requests
+/clients/:id/                                     : 1976 requests
+........
+
+Top 10 actions by time - cumulative
+========================================================================
+/overview/:date/                                  :   9044.582s [19359 requests]
+/overview/                                        :   8478.767s [3985 requests]
+/overview/:date/set/                              :   3309.041s [5589 requests]
+/clients/:id/products/:id/                        :   1479.911s [924 requests]
+/clients/:id/                                     :    750.080s [1976 requests]
+........
+
+Top 10 actions by time - per request mean
+========================================================================
+/overview/                                        :      2.128s [3985 requests]
+/clients/:id/products/:id/                        :      1.602s [924 requests]
+/overview/:date/set/                              :      0.592s [5589 requests]
+/overview/:date/                                  :      0.467s [19359 requests]
+/clients/:id/                                     :      0.380s [1976 requests]
+........
+
+Top 10 worst DB offenders - cumulative time
+========================================================================
+/overview/:date/                                  :   8773.993s [19359 requests]
+/overview/                                        :   8394.754s [3985 requests]
+/overview/:date/set/                              :   3307.928s [5589 requests]
+/clients/:id/products/:id/                        :   1425.220s [924 requests]
+/clients/:id/                                     :    535.229s [1976 requests]
+........
+
+Top 10 worst DB offenders - mean time
+========================================================================
+/overview/:id/:id/:id/print/                      :      6.994s [448 requests]
+/overview/                                        :      2.128s [3985 requests]
+/clients/:id/products/:id/                        :      1.602s [924 requests]
+/overview/:date/set/                              :      0.592s [5589 requests]
+/overview/:date/                                  :      0.467s [19359 requests]
+........
+
+Mongrel process blockers (> 1.0 seconds) - frequency
+========================================================================
+/overview/:date/                                  :   7494.233s [3144 requests]
+/overview/                                        :   8320.293s [1549 requests]
+/overview/:date/set/                              :   1149.235s [803 requests]
+/overview/:id/:id/:id/print/new/                  :    613.693s [341 requests]
+/clients/:id/products/:id/                        :   1370.693s [313 requests]
+........
+
+Requests graph - per hour
+========================================================================
+         ........
+         7:00 - 2731                 : XXXXXXX
+         8:00 - 6139                 : XXXXXXXXXXXXXXXX
+         9:00 - 7465                 : XXXXXXXXXXXXXXXXXXXX
+        10:00 - 7118                 : XXXXXXXXXXXXXXXXXXX
+        11:00 - 7409                 : XXXXXXXXXXXXXXXXXXX
+        12:00 - 6450                 : XXXXXXXXXXXXXXXXX
+        13:00 - 5377                 : XXXXXXXXXXXXXX
+        14:00 - 6058                 : XXXXXXXXXXXXXXXX
+        15:00 - 4156                 : XXXXXXXXXXX
+        16:00 - 2767                 : XXXXXXX
+        17:00 - 1598                 : XXXX
+        18:00 - 792                  : XX
+        ........

data/Rakefile

@@ -0,0 +1,43 @@
+require 'rubygems'
+
+load 'test/tasks.rake'
+ 
+desc 'Default: run unit tests.'
+task :default => :test
+
+
+namespace :gem do
+
+  desc "Builds a ruby gem for request-log-analyzer"
+  task :build => [:manifest] do
+    system %[gem build request-log-analyzer.gemspec]
+  end
+
+  desc %{Update ".manifest" with the latest list of project filenames. Respect\
+  .gitignore by excluding everything that git ignores. Update `files` and\
+  `test_files` arrays in "*.gemspec" file if it's present.}
+  task :manifest do
+    list = Dir['**/*'].sort
+    spec_file = Dir['*.gemspec'].first
+    list -= [spec_file] if spec_file
+  
+    File.read('.gitignore').each_line do |glob|
+      glob = glob.chomp.sub(/^\//, '')
+      list -= Dir[glob]
+      list -= Dir["#{glob}/**/*"] if File.directory?(glob) and !File.symlink?(glob)
+      puts "excluding #{glob}"
+    end
+ 
+    if spec_file
+      spec = File.read spec_file
+      spec.gsub! /^(\s* s.(test_)?files \s* = \s* )( \[ [^\]]* \] | %w\( [^)]* \) )/mx do
+        assignment = $1
+        bunch = $2 ? list.grep(/^test.*_test\.rb$/) : list
+        '%s%%w(%s)' % [assignment, bunch.join(' ')]
+      end
+      
+      File.open(spec_file,   'w') {|f| f << spec }
+    end
+    File.open('.manifest', 'w') {|f| f << list.join("\n") }
+  end
+end

data/TODO

@@ -0,0 +1,20 @@
+TODO items for Rails-log-analyzer
+=================================
+Contact willem AT vanbergen DOT org if you want to help out with the development.
+
+Summarizer:
+- Look at request types (GET / POST)
+
+Database:
+- Add query functionality for the resulting database file (interactive reports?)
+- Link request processing line to request completed line
+
+Rails integration:
+- Create script that calls request-log-analyzer 
+- Optionally use local or specific routes.rb file to parse URLs
+- Add rake tasks to Rails application when included
+
+General:
+- Add useful rake tasks
+- Add more tests
+- World domination

data/bin/request-log-analyzer

@@ -0,0 +1,95 @@
+#!/usr/bin/ruby
+require File.dirname(__FILE__) + '/../lib/command_line/arguments'
+require File.dirname(__FILE__) + '/../lib/rails_analyzer/log_parser'
+require File.dirname(__FILE__) + '/../lib/rails_analyzer/summarizer'
+require File.dirname(__FILE__) + '/../lib/bashcolorizer'
+require File.dirname(__FILE__) + '/../lib/ruby-progressbar/progressbar.rb'
+
+puts "Request log analyzer, by Willem van Bergen and Bart ten Brinke\n\n"
+
+# Substitutes variable elements in a url (like the id field) with a fixed string (like ":id")
+# This is used to aggregate simular requests.
+# <tt>request</tt> The request to evaluate.
+# Returns uniformed url string.
+# Raises on mailformed request.
+def request_hasher(request)
+  if request[:url]
+    url = request[:url].downcase.split(/^http[s]?:\/\/[A-z0-9\.-]+/).last.split('?').first # only the relevant URL part
+    url << '/' if url[-1] != '/'[0] && url.length > 1 # pad a trailing slash for consistency
+
+    url.gsub!(/\/\d+-\d+-\d+/, '/:date') # Combine all (year-month-day) queries
+    url.gsub!(/\/\d+-\d+/, '/:month') # Combine all date (year-month) queries
+    url.gsub!(/\/\d+/, '/:id') # replace identifiers in URLs
+        
+    return url
+  elsif request[:controller] && request[:action]
+    return "#{request[:controller]}##{request[:action]}"
+  else
+    raise 'Cannot hash this request! ' + request.inspect
+  end
+end
+
+# Print results using a ASCII table.
+# <tt>summarizer</tt> The summarizer containg information to draw the table.
+# <tt>field</tt> The field containing the data to be printed
+# <tt>amount</tt> The length of the table (defaults to 20)
+def print_table(summarizer, field, amount = 20)
+  summarizer.sort_actions_by(field).reverse[0, amount.to_i].each do |a|
+    # As we show count by default, show totaltime if we sort by count
+    field = :total_time if field == :count
+    
+    puts "#{a[0].ljust(50)}: %10.03fs [#{green("%d requests")}]" % [a[1][field], a[1][:count]]
+  end
+end
+
+# Parse the arguments given via commandline
+begin
+  $arguments = CommandLine::Arguments.parse do |command_line|
+    command_line.switch(:guess_database_time, :g)
+    command_line.switch(:fast, :f)
+    command_line.switch(:colorize, :z)
+    command_line.flag(:output, :alias => :o)
+    command_line.flag(:amount, :alias => :c)
+    command_line.required_files = 1
+  end
+  
+rescue CommandLine::Error => e
+  puts "ARGUMENT ERROR: " + e.message
+  puts
+  load File.dirname(__FILE__) + "/../output/usage.rb"
+  exit(0) 
+end
+
+$summarizer = RailsAnalyzer::Summarizer.new(:calculate_database => $arguments[:guess_database_time])
+$summarizer.blocker_duration = 1.0
+
+line_types = $arguments[:fast] ? [:completed] : [:started, :completed, :failed]
+
+# Walk through al the files given via the arguments.
+$arguments.files.each do |log_file|
+  puts "Processing #{line_types.join(', ')} log lines from #{log_file}..."
+  parser = RailsAnalyzer::LogParser.new(log_file)
+  
+  # add progress bar 
+  unless $arguments[:fast]
+    pbar = ProgressBar.new(green(log_file), File.size(log_file))
+    parser.progress { |pos, total| (pos == :finished) ? pbar.finish : pbar.set(pos) }
+  end
+  
+  parser.each(*line_types) do |request|
+    $summarizer.group(request)  { |r| request_hasher(r) }
+  end
+end
+
+# Select the reports to output and generate them.
+output_reports = $arguments[:output].split(',') rescue [:timespan, :most_requested, :total_time, :mean_time, :total_db_time, :mean_db_time, :mean_rendering_time, :blockers, :hourly_spread, :errors] 
+
+output_reports.each do |report|
+  report_location = "#{File.dirname(__FILE__)}/../output/#{report}.rb"
+
+  if File.exist?(report_location)
+    load report_location
+  else
+    puts "\nERROR: Output report #{report} not found!"
+  end
+end

data/bin/request-log-database

@@ -0,0 +1,79 @@
+#!/usr/bin/ruby
+
+require File.dirname(__FILE__) + '/../lib/command_line/arguments'
+require File.dirname(__FILE__) + '/../lib/rails_analyzer/log_parser'
+require File.dirname(__FILE__) + '/../lib/rails_analyzer/record_inserter'
+require File.dirname(__FILE__) + '/../lib/bashcolorizer'
+require File.dirname(__FILE__) + '/../lib/ruby-progressbar/progressbar.rb'
+
+
+puts "Rails log parser, by Willem van Bergen and Bart ten Brinke\n\n"
+
+begin
+  
+  $arguments = CommandLine::Arguments.parse do |command_line|
+    command_line.switch(:guess_database_time, :g)
+    command_line.switch(:reset_database, :r)
+    command_line.flag(:database, :alias => :d, :required => false)
+    command_line.required_files = 1
+  end
+  
+rescue CommandLine::Error => e
+  puts "ARGUMENT ERROR: " + e.message
+  puts
+  puts "Usage: ruby parsetodb.rb [LOGFILES*] <OPTIONS>"
+  puts
+  puts "Options:"
+  puts "  --database, -t:             The database file to use"
+  puts "  --reset-database, -r:       Resets the database before inserting new records"
+  puts "  --guess-database-time, -g:  Guesses the database duration of requests"      
+  puts
+  puts "Examples:"
+  puts "  ./parsetodb.rb development.log"
+  puts "  ./parsetodb.rb mongrel.0.log mongrel.1.log mongrel.2.log -g -d mongrel.db"  
+  puts 
+   
+  exit(0) 
+end
+
+log_files  = $arguments.files
+db_file    = $arguments[:database] || log_files.first + '.db'
+
+if $arguments[:reset_database] && File.exist?(db_file) 
+  File.delete(db_file) 
+  puts "Database file cleared."
+end
+
+records_inserted = 0 
+inserter = RailsAnalyzer::RecordInserter.insert_batch_into(db_file) do |db|
+  log_files.each do |log_file|
+
+    puts "Processing all log lines from #{log_file}..."
+    parser = RailsAnalyzer::LogParser.new(log_file)
+    
+    pbar = ProgressBar.new(green(log_file), File.size(log_file))
+    parser.progress { |pos, total| (pos == :finished) ? pbar.finish : pbar.set(pos) }
+    
+    parser.each do |request| 
+      db.insert(request) 
+      records_inserted += 1
+    end
+  end
+  
+  if $arguments[:guess_database_time]
+    puts "Calculating database times..."
+    db.calculate_db_durations! 
+  end
+end
+
+started   = inserter.count(:started)
+completed = inserter.count(:completed)
+failed    = inserter.count(:failed)
+
+puts 
+puts "Inserted #{records_inserted} records from #{log_files.length} files."
+puts "Parse warnings: #{inserter.warning_count}. Check the parse_warnings table in the database for details."
+puts
+puts "Requests started: #{started}"
+puts "Requests completed: #{completed}"
+puts "Requests failed: #{failed}"

data/lib/bashcolorizer.rb

@@ -0,0 +1,60 @@
+# Colorize a text output with the given color if.
+# <tt>text</tt> The text to colorize.
+# <tt>color_code</tt> The color code string to set
+# <tt>color</tt> Does not color if false. Defaults to ($arguments && $arguments[:colorize])
+def colorize(text, color_code, color = $arguments && $arguments[:colorize])
+  color ? "#{color_code}#{text}\e[0m" : text
+end
+
+# Draw a red line of text
+def red(text)
+  colorize(text, "\e[31m")
+end
+
+# Draw a Green line of text
+def green(text)
+  colorize(text, "\e[32m")
+end
+
+# Draw a Yellow line of text
+def yellow(text)
+  colorize(text, "\e[33m")
+end
+
+# Draw a Yellow line of text
+def blue(text)
+  colorize(text, "\e[34m")
+end
+
+def white(text)
+  colorize(text, "\e[37m")
+end
+
+
+#STYLE = {
+#      :default    =>    “33[0m”,
+#       # styles
+#       :bold       =>    “33[1m”,
+#       :underline  =>    “33[4m”,
+#       :blink      =>    “33[5m”,
+#       :reverse    =>    “33[7m”,
+#       :concealed  =>    “33[8m”,
+#      # font colors
+#       :black      =>    “33[30m”,
+#       :red        =>    “33[31m”,
+#       :green      =>    “33[32m”,
+#       :yellow     =>    “33[33m”,
+#       :blue       =>    “33[34m”,
+#       :magenta    =>    “33[35m”,
+#       :cyan       =>    “33[36m”,
+#       :white      =>    “33[37m”,
+#       # background colors
+#       :on_black   =>    “33[40m”,
+#       :on_red     =>    “33[41m”,
+#       :on_green   =>    “33[42m”,
+#       :on_yellow  =>    “33[43m”,
+#       :on_blue    =>    “33[44m”,
+#       :on_magenta =>    “33[45m”,
+#       :on_cyan    =>    “33[46m”,
+#       :on_white   =>    “33[47m” }
+#

data/lib/command_line/arguments.rb

@@ -0,0 +1,129 @@
+require "#{File.dirname(__FILE__)}/flag"
+require "#{File.dirname(__FILE__)}/exceptions"
+
+# Module used to parse commandline arguments
+module CommandLine
+  
+  # Parse argument lists and return an argument object containing all set flags and switches.
+  class Arguments
+    
+    FLAG_REGEXP = /^--?[A-z0-9]/
+    
+    attr_reader :flag_definitions
+    
+    attr_reader :flags
+    attr_reader :files
+    attr_reader :command
+    
+    attr_accessor :required_files
+    
+    # Initializer.
+    # <tt>arguments</tt> The arguments which are going to be parsed (defaults to $*).
+    def initialize(arguments = $*, &block)
+      @arguments = arguments
+      @flag_definitions = {}
+      @begins_with_command = false
+    end
+    
+    # Parse a list of arguments. Intatiates a Argument object with the given arguments and yeilds
+    # it so that flags and switches can be set by the application.
+    # <tt>arguments</tt> The arguments which are going to be parsed (defaults to $*).
+    # Returns the arguments object.parse!
+    def self.parse(arguments = $*, &block)
+      cla = Arguments.new(arguments)
+      yield(cla)
+      return cla.parse!
+    end
+    
+    # Handle argument switches for the application
+    # <tt>switch</tt> A switch symbol like :fast
+    # <tt>switch_alias</tt> An short alias for the same switch (:f).
+    def switch(switch, switch_alias = nil)
+      return self.flag(switch, :alias => switch_alias, :expects => nil)
+    end
+    
+    # Handle argument flags for the application
+    # <tt>flag</tt> A flag symbol like :fast
+    # Options
+    # * <tt>:expects</tt> Expects a value after the flag
+    def flag(flag, options)
+      options[:expects] = String unless options.has_key?(:expects)
+      argument = Flag.new(flag, options)
+      @flag_definitions[argument.to_argument]  = argument
+      @flag_definitions[argument.to_alias]     = argument if argument.has_alias?
+      return argument
+    end
+    
+    # If called argument list must begin with a command.
+    # <tt>begins_w_command</tt> Defaults to true.
+    def begins_with_command!(begins_w_command=true)
+      @begins_with_command = begins_w_command
+    end
+    
+    # Unknown flags will be silently ignored.
+    # <tt>ignore</tt> Defaults to true.
+    def ignore_unknown_flags!(ignore = true)
+      @ignore_unknown_flags = ignore
+    end
+  
+    def [](name)
+      return flags[name.to_s.gsub(/_/, '-').to_sym]
+    end
+  
+    # Parse the flags and switches set by the application.
+    # Returns an arguments object containing the flags and switches found in the commandline.
+    def parse!
+      @flags = {}
+      @files = []
+      
+      i = 0
+      while @arguments.length > i do
+        arg = @arguments[i]
+        if FLAG_REGEXP =~ arg
+          if @flag_definitions.has_key?(arg)
+            flag = @flag_definitions[arg]
+            if flag.expects_argument?
+              
+              if @arguments.length > (i + 1) && @arguments[i + 1]
+                @flags[flag.name] = @arguments[i + 1]
+                i += 1
+              else
+                raise CommandLine::FlagExpectsArgument.new(arg)
+              end              
+              
+            else
+              @flags[flag.name] = true
+            end
+          else
+            raise CommandLine::UnknownFlag.new(arg) unless @ignore_unknown_flags
+          end
+        else
+          if @begins_with_command && @command.nil?
+            @command = arg
+          else
+            @files << arg
+          end
+        end
+        i += 1        
+      end
+      
+      check_parsed_arguments!
+      
+      return self
+    end
+    
+    # Check if the parsed arguments meet their requirements.
+    # Raises CommandLineexception on error.
+    def check_parsed_arguments!
+      if @begins_with_command && @command.nil? 
+        raise CommandLine::CommandMissing.new
+      end
+
+      if @required_files && @files.length < @required_files
+        raise CommandLine::FileMissing.new("You need at least #{@required_files} files")
+      end
+
+    end
+  end
+
+end

data/lib/command_line/exceptions.rb

@@ -0,0 +1,37 @@
+module CommandLine
+  
+  # Commandline parsing errors and exceptions
+  class Error < Exception
+  end
+
+  # Missing a required flag
+  class FlagMissing < CommandLine::Error
+  end
+
+  # Missing a required file
+  class FileMissing < CommandLine::Error
+  end
+
+  # Missing a required flag argument
+  class FlagExpectsArgument < CommandLine::Error
+    def initialize(flag)
+      super("#{flag} expects an argument!")
+    end    
+  end
+
+  # Missing a required command
+  class CommandMissing < CommandLine::Error
+    def initialize(msg = "A command is missing")
+      super(msg)
+    end
+    
+  end
+
+  # Encountered an unkown flag
+  class UnknownFlag < CommandLine::Error
+    def initialize(flag)
+      super("#{flag} not recognized as a valid flag!")
+    end
+  end
+  
+end

data/lib/command_line/flag.rb

@@ -0,0 +1,51 @@
+module CommandLine
+  
+  # Argument flag handling.
+  class Flag
+    
+    attr_reader :name
+    attr_reader :alias
+    attr_reader :argument
+    
+    # Initialize new Flag
+    # <tt>name</tt> The name of the flag
+    # <tt>definition</tt> The definition of the flag.
+    def initialize(name, definition)
+      @name     = name.to_s.gsub(/_/, '-').to_sym
+      @alias    = definition[:alias].to_sym if definition[:alias]
+      @required = definition.has_key?(:required) && definition[:required] == true
+      @argument = definition[:expects] if definition[:expects]
+    end
+    
+    # Argument representation of the flag (--fast)
+    def to_argument
+      "--#{@name}"
+    end
+    
+    # Argument alias representation of the flag (-f)
+    def to_alias
+      "-#{@alias}"
+    end
+        
+    # Check if flag has an alias
+    def has_alias?
+      !@alias.nil?
+    end
+    
+    # Check if flag is optional
+    def optional?
+      !@required
+    end
+    
+    # Check if flag is required
+    def required?
+      @required
+    end
+    
+    # Check if flag expects an argument (Are you talking to me?)
+    def expects_argument?
+      !@argument.nil?
+    end
+  end  
+  
+end