wvanbergen-request-log-analyzer 1.0.0 → 1.0.1

data/DESIGN

@@ -0,0 +1,14 @@
+Request-log-analyzer is set up like a simple pipe and filter system.
+
+This allows you to easily add extra reports, filters and outputs.
+
+1) Build pipeline.
+                             -> Aggregator  (database)
+  Source -> Filter -> Filter -> Aggregator  (summary report)
+                             -> Aggregator  (...)
+
+2) Start chunk producer and push chunks through pipeline.
+  Controller.start
+
+3) Gather output from pipeline.
+  Controller.report

data/HACKING

@@ -0,0 +1,7 @@
+HACKING on r-l-a
+----------------
+
+- See DESIGN for the basic internal design of r-l-a
+- See http://wiki.github.com/wvanbergen/request-log-analyzer/development for
+  more information about developing
+- Contact willem AT vanbergen DOT org for any questions

data/README.textile

@@ -3,13 +3,18 @@ h1. Request log analyzer
 This is a simple command line tool to analyze request log files of both Rails and
 Merb. Its purpose is to find what actions are best candidates for optimization.
 
-* Analyzes Rails logs (all versions) and Merb logs
+* Analyzes Rails log files (all versions)
 * Can combine multiple files (handy if you are using logrotate)
-* Uses several metrics (cumulative time, average time, blockers, DB time, etc)
+* Uses several metrics, including cumulative request time, average request time, process blockers, database and rendering time, HTTP methods and states, Rails action cache statistics, etc.) ("Sample output":http://wiki.github.com/wvanbergen/request-log-analyzer/sample-output)
 * Low memory footprint (server-safe)
 * MIT licensed
 * Fast
 
+h2. Additional information
+
+* "Project wiki at GitHub":http://wiki.github.com/wvanbergen/request-log-analyzer
+* "wvanbergen's blog posts":http://techblog.floorplanner.com/tag/request-log-analyzer/
+
 h2. Installation
 
 @sudo gem install wvanbergen-request-log-analyzer --source http://gems.github.com@
@@ -30,10 +35,11 @@ h2. Usage
     --boring, -b               Output reports without ASCII colors.
     --database <filename>, -d: Creates an SQLite3 database of all the parsed request information.
     --debug                    Print debug information while parsing.
+    --file <filename>          Output to file.
 
   Examples:
     request-log-analyzer development.log
-    request-log-analyzer -z mongrel.0.log mongrel.1.log mongrel.2.log 
+    request-log-analyzer mongrel.0.log mongrel.1.log mongrel.2.log 
     request-log-analyzer --format merb -d requests.db production.log
 
   To install rake tasks in your Rails application, 
@@ -41,98 +47,3 @@ h2. Usage
 
     request-log-analyzer install rails
 </pre>
-
-
-h2. Example result
-
-Note that this example was shortened for your viewing pleasure.
-@$ request-log-analyzer /var/log/my_app.log -o 5@
-
-<pre>
-Request log analyzer, by Willem van Bergen and  Bart ten Brinke
-
-Processing started, failed, completed log lines from /var/log/my_app.log...
-
-Processing all log lines...
-========================================================================
-Timestamp first request: 2008-07-13T06:25:58+00:00
-Timestamp last request:  2008-07-20T06:18:53+00:00
-Total time analyzed: 7 days
-
-Total requests analyzed 58908 requests from log file
-Methods: GET (50.6%), POST (22.8%), PUT (25.4%), DELETE (1.1%), unknown (0.0%).
-
-Top 5 most requested actions
-========================================================================
-/overview/:date/                                  : 19359 requests
-/overview/day/:date/                              : 6365 requests
-/overview/:date/set/                              : 5589 requests
-/overview/                                        : 3985 requests
-/clients/:id/                                     : 1976 requests
-
-Top 5 actions by time - cumulative
-========================================================================
-/overview/:date/                                  :   9044.582s [19359 requests]
-/overview/                                        :   8478.767s [3985 requests]
-/overview/:date/set/                              :   3309.041s [5589 requests]
-/clients/:id/products/:id/                        :   1479.911s [924 requests]
-/clients/:id/                                     :    750.080s [1976 requests]
-
-Top 5 actions by time - per request mean
-========================================================================
-/overview/                                        :      2.128s [3985 requests]
-/clients/:id/products/:id/                        :      1.602s [924 requests]
-/overview/:date/set/                              :      0.592s [5589 requests]
-/overview/:date/                                  :      0.467s [19359 requests]
-/clients/:id/                                     :      0.380s [1976 requests]
-
-Top 5 worst DB offenders - cumulative time
-========================================================================
-/overview/:date/                                  :   8773.993s [19359 requests]
-/overview/                                        :   8394.754s [3985 requests]
-/overview/:date/set/                              :   3307.928s [5589 requests]
-/clients/:id/products/:id/                        :   1425.220s [924 requests]
-/clients/:id/                                     :    535.229s [1976 requests]
-
-Top 5 worst DB offenders - mean time
-========================================================================
-/overview/:id/:id/:id/print/                      :      6.994s [448 requests]
-/overview/                                        :      2.128s [3985 requests]
-/clients/:id/products/:id/                        :      1.602s [924 requests]
-/overview/:date/set/                              :      0.592s [5589 requests]
-/overview/:date/                                  :      0.467s [19359 requests]
-
-Mongrel process blockers (> 1.0 seconds)
-========================================================================
-/overview/:date/                                  :   7494.233s [3144 requests]
-/overview/                                        :   8320.293s [1549 requests]
-/overview/:date/set/                              :   1149.235s [803 requests]
-/overview/:id/:id/:id/print/new/                  :    613.693s [341 requests]
-/clients/:id/products/:id/                        :   1370.693s [313 requests]
-
-Requests graph - per hour
-========================================================================
-         ........
-         7:00 - 2731                 : XXXXXXX
-         8:00 - 6139                 : XXXXXXXXXXXXXXXX
-         9:00 - 7465                 : XXXXXXXXXXXXXXXXXXXX
-        10:00 - 7118                 : XXXXXXXXXXXXXXXXXXX
-        11:00 - 7409                 : XXXXXXXXXXXXXXXXXXX
-        12:00 - 6450                 : XXXXXXXXXXXXXXXXX
-        13:00 - 5377                 : XXXXXXXXXXXXXX
-        14:00 - 6058                 : XXXXXXXXXXXXXXXX
-        15:00 - 4156                 : XXXXXXXXXXX
-        16:00 - 2767                 : XXXXXXX
-        17:00 - 1598                 : XXXX
-        18:00 - 792                  : XX
-        ........
-
-Errors
-========================================================================
-StaleObjectError: [28 requests]
- -> Attempted to update a stale object
-StatementError: [2 requests]
- -> Mysql::Error: Deadlock found when trying to get lock; try restarting transaction
-NoMethodError: [1 requests]
- -> undefined method `code' for nil:NilClass
-</pre>

data/Rakefile

@@ -1,5 +1,5 @@
 Dir[File.dirname(__FILE__) + "/tasks/*.rake"].each { |file| load(file) }
  
-desc 'Default: run unit tests for request-log-analyzer.'
-task :default => :test
+desc 'Default: run RSpec for request-log-analyzer.'
+task :default => :spec
 

data/bin/request-log-analyzer

@@ -114,7 +114,7 @@ when :anonymize
   require File.dirname(__FILE__) + '/../lib/request_log_analyzer/log_processor'    
   RequestLogAnalyzer::LogProcessor.build(:anonymize, arguments).run!  
 else 
-  puts "Request log analyzer, by Willem van Bergen and Bart ten Brinke - Version 0.4.0\n\n"  
+  puts "Request log analyzer, by Willem van Bergen and Bart ten Brinke - Version 1.0\n\n"  
   
   # Run the request_log_analyzer!
   RequestLogAnalyzer::Controller.build(arguments, terminal_width).run!

data/lib/cli/bashcolorizer.rb

@@ -0,0 +1,60 @@
+# Colorize a text output with the given color if.
+# <tt>text</tt> The text to colorize.
+# <tt>color_code</tt> The color code string to set
+# <tt>color</tt> Does not color if false. Defaults to false
+def colorize(text, color_code, color = false)
+  color ? "#{color_code}#{text}\e[0m" : text
+end
+
+# Draw a red line of text
+def red(text, color = false)
+  colorize(text, "\e[31m", color)
+end
+
+# Draw a Green line of text
+def green(text, color = false)
+  colorize(text, "\e[32m", color)
+end
+
+# Draw a Yellow line of text
+def yellow(text, color = false)
+  colorize(text, "\e[33m", color)
+end
+
+# Draw a Yellow line of text
+def blue(text, color = false)
+  colorize(text, "\e[34m", color)
+end
+
+def white(text, color = false)
+  colorize(text, "\e[37m", color)
+end
+
+
+#STYLE = {
+#      :default    =>    “33[0m”,
+#       # styles
+#       :bold       =>    “33[1m”,
+#       :underline  =>    “33[4m”,
+#       :blink      =>    “33[5m”,
+#       :reverse    =>    “33[7m”,
+#       :concealed  =>    “33[8m”,
+#      # font colors
+#       :black      =>    “33[30m”,
+#       :red        =>    “33[31m”,
+#       :green      =>    “33[32m”,
+#       :yellow     =>    “33[33m”,
+#       :blue       =>    “33[34m”,
+#       :magenta    =>    “33[35m”,
+#       :cyan       =>    “33[36m”,
+#       :white      =>    “33[37m”,
+#       # background colors
+#       :on_black   =>    “33[40m”,
+#       :on_red     =>    “33[41m”,
+#       :on_green   =>    “33[42m”,
+#       :on_yellow  =>    “33[43m”,
+#       :on_blue    =>    “33[44m”,
+#       :on_magenta =>    “33[45m”,
+#       :on_cyan    =>    “33[46m”,
+#       :on_white   =>    “33[47m” }
+#

data/lib/cli/command_line_arguments.rb

@@ -0,0 +1,301 @@
+module CommandLine
+
+  class Option
+
+    attr_reader :name, :alias
+    attr_reader :parameter_count
+    attr_reader :default_value
+
+    # Rewrites a command line keyword by replacing the underscores with dashes
+    # <tt>sym</tt> The symbol to rewrite
+    def self.rewrite(sym)
+      sym.to_s.gsub(/_/, '-').to_sym
+    end
+
+    # Initialize new CommandLine::Option
+    # <tt>name</tt> The name of the flag
+    # <tt>definition</tt> The definition of the flag.
+    def initialize(name, definition = {})
+      @name            = CommandLine::Option.rewrite(name)
+      @alias           = definition[:alias].to_sym if definition[:alias]
+      @required        = definition.has_key?(:required) && definition[:required] == true
+      @parameter_count = definition[:parameters] || 1
+      @multiple        = definition[:multiple]   || false
+      @default_value   = definition[:default]    || false
+    end
+
+    def parse(arguments_parser)
+      if @parameter_count == 0
+        return true
+      elsif @parameter_count == 1
+        parameter = arguments_parser.next_parameter
+        raise CommandLine::ParameterExpected, self if parameter.nil?
+        return parameter
+      elsif @parameter_count == :any
+        parameters = []
+        while parameter = arguments_parser.next_parameter && parameter != '--'
+          parameters << parameter
+        end
+        return parameters
+      else
+        parameters = []
+        @parameter_count.times do |n|
+          parameter = arguments_parser.next_parameter
+          raise CommandLine::ParameterExpected, self if parameter.nil?
+          parameters << parameter
+        end
+        return parameters
+      end
+    end
+
+    def =~(test)
+      [@name, @alias].include?(CommandLine::Option.rewrite(test))
+    end
+
+    # Argument representation of the flag (--fast)
+    def to_option
+      "--#{@name}"
+    end
+
+    # Argument alias representation of the flag (-f)
+    def to_alias
+      "-#{@alias}"
+    end
+
+    # Check if flag has an alias
+    def has_alias?
+      !@alias.nil?
+    end
+
+    # Check if flag is required
+    def required?
+      @required
+    end
+    
+    # Check if flag is optional
+    def optional?
+      !@required
+    end
+
+    def multiple?
+      @multiple
+    end
+
+    def has_default?
+      !@default_value.nil?
+    end
+  end
+  
+  class Arguments
+
+    class Definition
+      
+      ENDLESS_PARAMETERS = 99999
+      
+      attr_reader :commands, :options, :parameters
+      
+      def initialize(parent)
+        @parent = parent
+        @options  = {}
+        @commands = {}
+        @parameters = nil
+      end
+      
+      def [](option_name)
+        option_symbol = CommandLine::Option.rewrite(option_name)
+        if the_option = @options.detect { |(name, odef)| odef =~ option_symbol }
+          the_option[1]
+        else
+          raise CommandLine::UnknownOption, option_name
+        end
+      end
+      
+      def minimum_parameters=(count_specifier)
+        @parameters = count_specifier..ENDLESS_PARAMETERS
+      end
+            
+      def parameters=(count_specifier)
+        @parameters = count_specifier
+      end
+      
+      alias :files= :parameters= 
+      
+      def option(name, options = {})
+        clo = CommandLine::Option.new(name, options)
+        @options[clo.name] = clo
+      end
+      
+      def switch(name, switch_alias = nil)
+        option(name, :alias => switch_alias, :parameters => 0)
+      end
+
+      def command(name, &block)
+        command_definition = Definition.new(self)
+        yield(command_definition) if block_given?
+        @commands[CommandLine::Option.rewrite(name)] = command_definition
+      end
+      
+      def has_command?(command)
+        @commands[CommandLine::Option.rewrite(command)]
+      end
+    end
+    
+    OPTION_REGEXP  = /^\-\-([A-z0-9-]+)$/;
+    ALIASES_REGEXP = /^\-([A-z0-9]+)$/
+    
+    attr_reader :definition
+    attr_reader :tokens
+    attr_reader :command, :options, :parameters
+    
+    def self.parse(tokens = $*, &block)
+      cla = Arguments.new
+      cla.define(&block)
+      return cla.parse!(tokens)   
+    end
+    
+    def initialize
+      @tokens = []
+      @definition = Definition.new(self)    
+      @current_definition = @definition        
+    end
+    
+    def define(&block)
+      yield(@definition)
+    end
+    
+    def [](option)
+      if the_option = @options.detect { |(key, value)| key =~ option }
+        the_option[1]
+      else
+        @current_definition[option].default_value
+      end
+    end
+        
+    def next_token
+      @current_token = @tokens.shift
+      return @current_token
+    end
+    
+    def next_parameter
+      parameter_candidate = @tokens.first
+      parameter = (parameter_candidate.nil? || OPTION_REGEXP =~ parameter_candidate || ALIASES_REGEXP =~ parameter_candidate) ? nil : @tokens.shift
+      return parameter
+    end
+    
+    def parse!(tokens)
+      @current_definition = @definition         
+      @first_token = true
+      @tokens      = tokens.clone
+      
+      @options = {}
+      @parameters   = []
+      @command = nil
+      
+      prepare_result!
+      
+      while next_token
+        
+        if @first_token && command_definition = @definition.has_command?(@current_token)
+          @current_definition = command_definition
+          @command = CommandLine::Option.rewrite(@current_token)
+        else
+          case @current_token
+            when ALIASES_REGEXP; handle_alias_expansion($1)
+            when OPTION_REGEXP;  handle_option($1)
+            else;                handle_other_parameter(@current_token)
+          end
+          @first_token = false          
+        end
+          
+      end
+      
+      validate_arguments!
+      
+      return self     
+    end
+    
+    protected
+    
+    def prepare_result!
+      multiple_options = Hash[*@current_definition.options.select { |name, o| o.multiple? }.flatten]
+      multiple_options.each { |name, definition| @options[definition] = [] }
+    end
+    
+    def validate_arguments!
+      if @current_definition.parameters && !(@current_definition.parameters === @parameters.length)
+        raise CommandLine::ParametersOutOfRange.new(@current_definition.parameters, @parameters.length)
+      end
+      
+      required_options = Hash[*@current_definition.options.select { |name, o| o.required? }.flatten]
+      required_options.each do |name, definition|
+        raise CommandLine::RequiredOptionMissing, definition unless self[name] 
+      end
+    end
+    
+    def handle_alias_expansion(aliases)
+      aliases.reverse.scan(/./) do |alias_char|
+        if option_definition = @current_definition[alias_char]
+          @tokens.unshift(option_definition.to_option)
+        else
+          raise CommandLine::UnknownOption, alias_char
+        end
+      end
+    end
+    
+    def handle_other_parameter(parameter)
+      @parameters << parameter
+    end
+    
+    def handle_option(option_name)
+      option_definition = @current_definition[option_name]
+      raise CommandLine::UnknownOption, option_name if option_definition.nil?
+      
+      if option_definition.multiple?
+        @options[option_definition] << option_definition.parse(self)        
+      else
+        @options[option_definition] = option_definition.parse(self)
+      end
+    end
+        
+  end
+  
+  # Commandline parsing errors and exceptions
+  class Error < Exception
+  end
+
+  # Missing a required flag
+  class RequiredOptionMissing < CommandLine::Error
+    def initialize(option)
+      super("You have to provide the #{option.name} option!")
+    end    
+  end
+
+  # Missing a required file
+  class ParametersOutOfRange < CommandLine::Error
+    def initialize(expected, actual)
+      if expected.kind_of?(Range)
+        if expected.end == CommandLine::Arguments::Definition::ENDLESS_PARAMETERS
+          super("The command expected at least #{expected.begin} parameters, but found #{actual}!")
+        else
+          super("The command expected between #{expected.begin} and #{expected.end} parameters, but found #{actual}!")
+        end
+      else
+        super("The command expected #{expected} parameters, but found #{actual}!")
+      end
+    end    
+  end
+
+  # Missing a required flag argument
+  class ParameterExpected < CommandLine::Error
+    def initialize(option)
+      super("The option #{option.inspect} expects a parameter!")
+    end    
+  end
+
+  # Encountered an unkown flag
+  class UnknownOption < CommandLine::Error
+    def initialize(option_identifier)
+      super("#{option_identifier.inspect} not recognized as a valid option!")
+    end
+  end  
+end