ngmoco-request-log-analyzer 1.4.2

data/.gitignore

@@ -0,0 +1,10 @@
+.svn/
+.DS_Store
+request-log-analyzer-*.gem
+requests.db
+/pkg
+/doc
+/tmp
+/classes
+/files
+/coverage

data/DESIGN.rdoc

@@ -0,0 +1,41 @@
+=== Request-log-analyzer
+RLA is set up like a simple pipe and filter system.
+
+This allows you to easily add extra reports, filters and outputs.
+                             -> Aggregator  (database)
+  Source -> Filter -> Filter -> Aggregator  (summary report)  -> Output
+                             -> Aggregator  (...)
+
+When the pipeline has been constructed, we Start chunk producer (source) and push requests through pipeline.
+
+  Controller.start
+
+=== Source
+RequestLogAnalyzer::Source is an Object that pushes requests into the chain.
+At the moment you can only use the log-parser as a source.
+It accepts files or stdin and can parse then into request objects using a RequestLogAnalyzer::FileFormat definition.
+In the future we want to be able to have a generated request database as source as this will make interactive
+down drilling possible.
+
+=== Filter
+The filters are all subclasses of the RequestLogAnalyzer::Filter class.
+They accept a request object, manipulate or drop it, and then pass the request object on to the next filter
+in the chain.
+At the moment there are three types of filters available: Anonymize, Field and Timespan.
+
+=== Aggregator
+The Aggregators all inherit from the RequestLogAnalyzer::Aggregator class.
+All the requests that come out of the Filterchain are fed into all the aggregators in parallel.
+These aggregators can do anything what they want with the given request.
+For example: the Database aggregator will just store all the requests into a SQLite database while the Summarizer will
+generate a wide range of statistical reports from them.
+
+=== Running the pipeline
+All Aggregators are asked to report what they have done. For example the database will report: I stuffed x requests
+into SQLite database Y. The Summarizer will output its reports.
+
+  Controller.report
+
+The output is pushed to a RequestLogAnalyzer::Output object, which takes care of the output.
+It can generate either ASCII, UTF8 or even HTML output.
+

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2008-2009 Willem van Bergen & Bart ten Brinke
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.rdoc

@@ -0,0 +1,39 @@
+= Request-log-analyzer
+
+This is a simple command line tool to analyze request log files in various formats to produce a performance report. Its purpose is to find what actions are best candidates for optimization.
+
+* Analyzes Rails request logs, Merb request logs, Apache access logs and more, or parses any other log format you specify.
+* Combines multiple files and decompresses compressed files, which comes in handy if you are using logrotate.
+* Uses several metrics, including cumulative request time, mean request time, process blockers, database and rendering time, HTTP methods and statuses, Rails action cache statistics, etc.) (Sample output: http://wiki.github.com/wvanbergen/request-log-analyzer/sample-output)
+* Low memory footprint and reasonably fast, so it is safe to run on a production server.
+* MIT licensed
+
+See the project wiki at http://wiki.github.com/wvanbergen/request-log-analyzer for documentation and additional information.
+
+== Installation & basic usage
+
+Install request-log-analyzer as a Ruby gem (you might need to run this command
+as root by prepending +sudo+ to it):
+
+  $ gem install request-log-analyzer
+
+To analyze a Rails log file and produce a performance report, run
+request-log-analyzer like this:
+
+  $ request-log-analyzer log/production.log
+
+For more details, other file formats, and available command line options, see the project's wiki at http://wiki.github.com/wvanbergen/request-log-analyzer
+
+== Additional information
+
+Request-log-analyzer was designed and built by Willem van Bergen and Bart ten
+Brinke.
+
+Do you have a rails application that is not performing as it should? If you need
+an expert to analyze your application, feel free to contact either Willem van
+Bergen (willem@railsdoctors.com) or Bart ten Brinke (bart@railsdoctors.com).
+
+* Project wiki at GitHub: http://wiki.github.com/wvanbergen/request-log-analyzer
+* railsdoctors homepage: http://railsdoctors.com
+* wvanbergen's blog posts: http://techblog.floorplanner.com/tag/request-log-analyzer
+* barttenbrinke's blog posts: http://movesonrails.com/articles/tag/analyzer

data/Rakefile

@@ -0,0 +1,8 @@
+Dir[File.dirname(__FILE__) + "/tasks/*.rake"].each { |file| load(file) }
+
+# Create rake tasks for a gem manages by github. The tasks are created in the
+# gem namespace
+GithubGem::RakeTasks.new(:gem)
+
+# Set the RSpec runner with specdoc output as default task.
+task :default => "spec:specdoc"

data/bin/request-log-analyzer

@@ -0,0 +1,114 @@
+#!/usr/bin/env ruby
+# encoding: utf-8
+
+$:.unshift(File.dirname(__FILE__) + '/../lib')
+require 'request_log_analyzer'
+require 'cli/command_line_arguments'
+require 'cli/progressbar'
+require 'cli/tools'
+
+# Parse the arguments given via commandline
+begin
+  arguments = CommandLine::Arguments.parse do |command_line|
+
+    command_line.command(:install) do |install|
+      install.parameters = 1
+    end
+
+    command_line.command(:console) do |cons|
+      cons.option(:database, :alias => :d, :required => true)
+    end
+
+    command_line.command(:strip) do |strip|
+      strip.minimum_parameters = 1
+      strip.option(:format, :alias => :f, :default => 'rails')
+      strip.option(:output, :alias => :o)
+      strip.switch(:discard_teaser_lines, :t)
+      strip.switch(:keep_junk_lines, :j)
+    end
+
+    command_line.option(:format, :alias => :f, :default => 'rails')
+    command_line.option(:apache_format)
+    command_line.option(:rails_format)
+
+    command_line.option(:file, :alias => :e)
+    command_line.option(:mail, :alias => :m)
+    command_line.option(:parse_strategy, :default => 'assume-correct')
+    command_line.option(:dump)
+
+    command_line.option(:aggregator, :alias => :a, :multiple => true)
+
+    command_line.option(:database, :alias => :d)
+    command_line.switch(:reset_database)
+
+    # filtering options
+    command_line.option(:select, :multiple => true, :parameters => 2)
+    command_line.option(:reject, :multiple => true, :parameters => 2)
+    command_line.option(:after)
+    command_line.option(:before)
+
+    command_line.switch(:boring, :b)
+    command_line.option(:output, :alias => :o, :default => 'FixedWidth')
+    command_line.option(:report_width,  :default => terminal_width - 1)
+    command_line.option(:report_amount, :default => 20)
+    command_line.option(:report_sort,   :default => 'sum,mean')
+
+    command_line.switch(:debug)
+    command_line.switch(:no_progress)
+
+    command_line.minimum_parameters = 1
+  end
+
+rescue CommandLine::Error => e
+  puts "Request-log-analyzer, by Willem van Bergen and Bart ten Brinke - version #{RequestLogAnalyzer::VERSION}"
+  puts "Website: http://railsdoctors.com"
+  puts
+  puts "ARGUMENT ERROR: " + e.message if e.message
+  puts
+  puts "Usage: request-log-analyzer [LOGFILES*] <OPTIONS>"
+  puts
+  puts "Input options:"
+  puts "  --format <format>, -f:     Uses the specified log file format. Defaults to rails."
+  puts "  --after <date>             Only consider requests from <date> or later."
+  puts "  --before <date>            Only consider requests before <date>."
+  puts "  --select <field> <value>   Only consider requests where <field> matches <value>."
+  puts "  --reject <field> <value>   Only consider requests where <field> does not match <value>."
+  puts
+  puts "Output options:"
+  puts "  --boring, -b               Output reports without ASCII colors."
+  puts "  --database <filename>, -d: Creates an SQLite3 database of all the parsed request information."
+  puts "  --debug                    Print debug information while parsing."
+  puts "  --file <filename>          Output to file."
+  puts "  --mail <emailaddress>      Send report to an email address."
+  puts "  --output <format>          Output format. Supports 'HTML' and 'FixedWidth' (default)"
+  puts "  --dump <filename>          Dump the YAML formatted results in the given file"
+  puts
+  puts "Examples:"
+  puts "  request-log-analyzer development.log"
+  puts "  request-log-analyzer -b mongrel.0.log mongrel.1.log mongrel.2.log "
+  puts "  request-log-analyzer --format merb -d requests.db production.log"
+  puts
+  puts "To install rake tasks in your Rails application, "
+  puts "run the following command in your application's root directory:"
+  puts
+  puts "  request-log-analyzer install rails"
+  exit(0)
+end
+
+case arguments.command
+when :install
+  install_rake_tasks(arguments.parameters[0])
+when :console
+  require 'cli/database_console'
+  DatabaseConsole.new(arguments).run!
+when :strip
+  require File.dirname(__FILE__) + '/../lib/request_log_analyzer/log_processor'
+  RequestLogAnalyzer::LogProcessor.build(:strip, arguments).run!
+else
+  puts "Request-log-analyzer, by Willem van Bergen and Bart ten Brinke - version #{RequestLogAnalyzer::VERSION}"
+  puts "Website: http://railsdoctors.com"
+  puts
+
+  # Run the request_log_analyzer!
+  RequestLogAnalyzer::Controller.build_from_arguments(arguments).run!
+end

data/lib/cli/command_line_arguments.rb

@@ -0,0 +1,301 @@
+module CommandLine
+
+  class Option
+
+    attr_reader :name, :alias
+    attr_reader :parameter_count
+    attr_reader :default_value
+
+    # Rewrites a command line keyword by replacing the underscores with dashes
+    # <tt>sym</tt> The symbol to rewrite
+    def self.rewrite(sym)
+      sym.to_s.gsub(/_/, '-').to_sym
+    end
+
+    # Initialize new CommandLine::Option
+    # <tt>name</tt> The name of the flag
+    # <tt>definition</tt> The definition of the flag.
+    def initialize(name, definition = {})
+      @name            = CommandLine::Option.rewrite(name)
+      @alias           = definition[:alias].to_sym if definition[:alias]
+      @required        = definition.has_key?(:required) && definition[:required] == true
+      @parameter_count = definition[:parameters] || 1
+      @multiple        = definition[:multiple]   || false
+      @default_value   = definition[:default]    || false
+    end
+
+    def parse(arguments_parser)
+      if @parameter_count == 0
+        return true
+      elsif @parameter_count == 1
+        parameter = arguments_parser.next_parameter
+        raise CommandLine::ParameterExpected, self if parameter.nil?
+        return parameter
+      elsif @parameter_count == :any
+        parameters = []
+        while parameter = arguments_parser.next_parameter && parameter != '--'
+          parameters << parameter
+        end
+        return parameters
+      else
+        parameters = []
+        @parameter_count.times do |n|
+          parameter = arguments_parser.next_parameter
+          raise CommandLine::ParameterExpected, self if parameter.nil?
+          parameters << parameter
+        end
+        return parameters
+      end
+    end
+
+    def =~(test)
+      [@name, @alias].include?(CommandLine::Option.rewrite(test))
+    end
+
+    # Argument representation of the flag (--fast)
+    def to_option
+      "--#{@name}"
+    end
+
+    # Argument alias representation of the flag (-f)
+    def to_alias
+      "-#{@alias}"
+    end
+
+    # Check if flag has an alias
+    def has_alias?
+      !@alias.nil?
+    end
+
+    # Check if flag is required
+    def required?
+      @required
+    end
+
+    # Check if flag is optional
+    def optional?
+      !@required
+    end
+
+    def multiple?
+      @multiple
+    end
+
+    def has_default?
+      !@default_value.nil?
+    end
+  end
+
+  class Arguments
+
+    class Definition
+
+      ENDLESS_PARAMETERS = 99999
+
+      attr_reader :commands, :options, :parameters
+
+      def initialize(parent)
+        @parent = parent
+        @options  = {}
+        @commands = {}
+        @parameters = nil
+      end
+
+      def [](option_name)
+        option_symbol = CommandLine::Option.rewrite(option_name)
+        if the_option = @options.detect { |(name, odef)| odef =~ option_symbol }
+          the_option[1]
+        else
+          raise CommandLine::UnknownOption, option_name
+        end
+      end
+
+      def minimum_parameters=(count_specifier)
+        @parameters = count_specifier..ENDLESS_PARAMETERS
+      end
+
+      def parameters=(count_specifier)
+        @parameters = count_specifier
+      end
+
+      alias :files= :parameters=
+
+      def option(name, options = {})
+        clo = CommandLine::Option.new(name, options)
+        @options[clo.name] = clo
+      end
+
+      def switch(name, switch_alias = nil)
+        option(name, :alias => switch_alias, :parameters => 0)
+      end
+
+      def command(name, &block)
+        command_definition = Definition.new(self)
+        yield(command_definition) if block_given?
+        @commands[CommandLine::Option.rewrite(name)] = command_definition
+      end
+
+      def has_command?(command)
+        @commands[CommandLine::Option.rewrite(command)]
+      end
+    end
+
+    OPTION_REGEXP  = /^\-\-([A-Za-z0-9-]+)$/;
+    ALIASES_REGEXP = /^\-([A-Aa-z0-9]+)$/
+
+    attr_reader :definition
+    attr_reader :tokens
+    attr_reader :command, :options, :parameters
+
+    def self.parse(tokens = $*, &block)
+      cla = Arguments.new
+      cla.define(&block)
+      return cla.parse!(tokens)
+    end
+
+    def initialize
+      @tokens = []
+      @definition = Definition.new(self)
+      @current_definition = @definition
+    end
+
+    def define(&block)
+      yield(@definition)
+    end
+
+    def [](option)
+      if the_option = @options.detect { |(key, value)| key =~ option }
+        the_option[1]
+      else
+        @current_definition[option].default_value
+      end
+    end
+
+    def next_token
+      @current_token = @tokens.shift
+      return @current_token
+    end
+
+    def next_parameter
+      parameter_candidate = @tokens.first
+      parameter = (parameter_candidate.nil? || OPTION_REGEXP =~ parameter_candidate || ALIASES_REGEXP =~ parameter_candidate) ? nil : @tokens.shift
+      return parameter
+    end
+
+    def parse!(tokens)
+      @current_definition = @definition
+      @first_token = true
+      @tokens      = tokens.clone
+
+      @options = {}
+      @parameters   = []
+      @command = nil
+
+      prepare_result!
+
+      while next_token
+
+        if @first_token && command_definition = @definition.has_command?(@current_token)
+          @current_definition = command_definition
+          @command = CommandLine::Option.rewrite(@current_token)
+        else
+          case @current_token
+            when ALIASES_REGEXP; handle_alias_expansion($1)
+            when OPTION_REGEXP;  handle_option($1)
+            else;                handle_other_parameter(@current_token)
+          end
+          @first_token = false
+        end
+
+      end
+
+      validate_arguments!
+
+      return self
+    end
+
+    protected
+
+    def prepare_result!
+      multiple_options = Hash[*@current_definition.options.select { |name, o| o.multiple? }.flatten]
+      multiple_options.each { |name, definition| @options[definition] = [] }
+    end
+
+    def validate_arguments!
+      if @current_definition.parameters && !(@current_definition.parameters === @parameters.length)
+        raise CommandLine::ParametersOutOfRange.new(@current_definition.parameters, @parameters.length)
+      end
+
+      required_options = Hash[*@current_definition.options.select { |name, o| o.required? }.flatten]
+      required_options.each do |name, definition|
+        raise CommandLine::RequiredOptionMissing, definition unless self[name]
+      end
+    end
+
+    def handle_alias_expansion(aliases)
+      aliases.reverse.scan(/./) do |alias_char|
+        if option_definition = @current_definition[alias_char]
+          @tokens.unshift(option_definition.to_option)
+        else
+          raise CommandLine::UnknownOption, alias_char
+        end
+      end
+    end
+
+    def handle_other_parameter(parameter)
+      @parameters << parameter
+    end
+
+    def handle_option(option_name)
+      option_definition = @current_definition[option_name]
+      raise CommandLine::UnknownOption, option_name if option_definition.nil?
+
+      if option_definition.multiple?
+        @options[option_definition] << option_definition.parse(self)
+      else
+        @options[option_definition] = option_definition.parse(self)
+      end
+    end
+
+  end
+
+  # Commandline parsing errors and exceptions
+  class Error < Exception
+  end
+
+  # Missing a required flag
+  class RequiredOptionMissing < CommandLine::Error
+    def initialize(option)
+      super("You have to provide the #{option.name} option!")
+    end
+  end
+
+  # Missing a required file
+  class ParametersOutOfRange < CommandLine::Error
+    def initialize(expected, actual)
+      if expected.kind_of?(Range)
+        if expected.end == CommandLine::Arguments::Definition::ENDLESS_PARAMETERS
+          super("The command expected at least #{expected.begin} parameters, but found #{actual}!")
+        else
+          super("The command expected between #{expected.begin} and #{expected.end} parameters, but found #{actual}!")
+        end
+      else
+        super("The command expected #{expected} parameters, but found #{actual}!")
+      end
+    end
+  end
+
+  # Missing a required flag argument
+  class ParameterExpected < CommandLine::Error
+    def initialize(option)
+      super("The option #{option.inspect} expects a parameter!")
+    end
+  end
+
+  # Encountered an unkown flag
+  class UnknownOption < CommandLine::Error
+    def initialize(option_identifier)
+      super("#{option_identifier.inspect} not recognized as a valid option!")
+    end
+  end
+end