wvanbergen-request-log-analyzer 1.2.0 → 1.2.1

data/{DESIGN → DESIGN.rdoc}

@@ -1,38 +1,41 @@
-Request-log-analyzer is set up like a simple pipe and filter system.
+=== Request-log-analyzer
+RLA is set up like a simple pipe and filter system.
 
 This allows you to easily add extra reports, filters and outputs.
-
-1) Build pipeline.
                              -> Aggregator  (database)
   Source -> Filter -> Filter -> Aggregator  (summary report)  -> Output
                              -> Aggregator  (...)
-                             
-2) Start chunk producer and push chunks through pipeline.
- Controller.start
- 
+
+When the pipeline has been constructed, we Start chunk producer (source) and push requests through pipeline.
+
+  Controller.start
+
+=== Source
 RequestLogAnalyzer::Source is an Object that pushes requests into the chain.
 At the moment you can only use the log-parser as a source.
 It accepts files or stdin and can parse then into request objects using a RequestLogAnalyzer::FileFormat definition.
 In the future we want to be able to have a generated request database as source as this will make interactive
 down drilling possible.
 
+=== Filter
 The filters are all subclasses of the RequestLogAnalyzer::Filter class.
 They accept a request object, manipulate or drop it, and then pass the request object on to the next filter
 in the chain.
 At the moment there are three types of filters available: Anonymize, Field and Timespan.
 
+=== Aggregator
 The Aggregators all inherit from the RequestLogAnalyzer::Aggregator class.
 All the requests that come out of the Filterchain are fed into all the aggregators in parallel.
 These aggregators can do anything what they want with the given request.
 For example: the Database aggregator will just store all the requests into a SQLite database while the Summarizer will
 generate a wide range of statistical reports from them.
 
-3) Gather output from pipeline.
-  Controller.report
-
+=== Running the pipeline
 All Aggregators are asked to report what they have done. For example the database will report: I stuffed x requests
 into SQLite database Y. The Summarizer will output its reports.
 
+  Controller.report
+
 The output is pushed to a RequestLogAnalyzer::Output object, which takes care of the output.
 It can generate either ASCII, UTF8 or even HTML output.
 

data/README.rdoc

@@ -3,11 +3,10 @@
 This is a simple command line tool to analyze request log files of both Rails and
 Merb to produce a performance report. Its purpose is to find what actions are best candidates for optimization.
 
-* Analyzes Rails log files (all versions), Merb logs, or any other log format
-* Can combine multiple files (handy if you are using logrotate)
+* Analyzes Rails log files (all versions), Merb logs, or any other log format you specify
+* Combines multiple files and decompresses compressed files (handy if you are using logrotate)
 * Uses several metrics, including cumulative request time, average request time, process blockers, database and rendering time, HTTP methods and statuses, Rails action cache statistics, etc.) (Sample output: http://wiki.github.com/wvanbergen/request-log-analyzer/sample-output)
-* Low memory footprint (server-safe)
-* Fast
+* Low memory footprint and reasonably fast (server-safe)
 * MIT licensed
 
 Request log analyzer was designed and built by Willem van Bergen and Bart ten Brinke.
@@ -36,7 +35,10 @@ For more details and available command line options, see the project's wiki: htt
 
 == Additional information
 
+Do you have a rails application that is not performing as it should?
+If you need an expert to analyze your application, feel free to contact either Willem van Bergen (willem@railsdoctors.com) or Bart ten Brinke (bart@railsdoctors.com).
+
 * Project wiki at GitHub: http://wiki.github.com/wvanbergen/request-log-analyzer
-* RDoc documentation: http://wvanbergen.github.com/request-log-analyzer
+* railsdoctors homepage: http://railsdoctors.com
 * wvanbergen's blog posts: http://techblog.floorplanner.com/tag/request-log-analyzer
 * barttenbrinke's blog posts: http://movesonrails.com/articles/tag/analyzer

data/RELEASE_NOTES.rdoc

@@ -0,0 +1,10 @@
+=== 1.2.1
+ * Compressed logfile support
+ * Parsable YAML results as output (use --dump <filename> )
+ * Full Rdoc
+ * Release notes
+ * Updated rails take tasks
+
+=== 1.2.0
+ * Ruby 1.9 support
+ * Rcov coverage 

data/bin/request-log-analyzer

@@ -17,13 +17,14 @@ begin
       strip.option(:format, :alias => :f, :default => 'rails')      
       strip.option(:output, :alias => :o)
       strip.switch(:discard_teaser_lines, :t)
-      strip.switch(:keep_junk_lines, :j)      
+      strip.switch(:keep_junk_lines, :j) 
     end
     
     command_line.option(:format, :alias => :f, :default => 'rails')
     command_line.option(:file, :alias => :e)
     command_line.option(:parse_strategy, :default => 'assume-correct')
-    
+    command_line.option(:dump)
+          
     command_line.option(:aggregator, :alias => :a, :multiple => true)
     command_line.option(:database, :alias => :d)    
 
@@ -44,6 +45,8 @@ begin
   
 rescue CommandLine::Error => e
   puts "Request-log-analyzer, by Willem van Bergen and Bart ten Brinke - version #{RequestLogAnalyzer::VERSION}"  
+  puts "Website: http://railsdoctors.com"
+  puts
   puts "ARGUMENT ERROR: " + e.message if e.message
   puts
   puts "Usage: request-log-analyzer [LOGFILES*] <OPTIONS>"
@@ -61,6 +64,7 @@ rescue CommandLine::Error => e
   puts "  --debug                    Print debug information while parsing."
   puts "  --file <filename>          Output to file."
   puts "  --output <format>          Output format. Supports 'HTML' and 'FixedWidth' (default)"  
+  puts "  --dump <filename>          Dump the YAML formatted results in the given file"
   puts 
   puts "Examples:"
   puts "  request-log-analyzer development.log"
@@ -82,7 +86,7 @@ when :strip
   RequestLogAnalyzer::LogProcessor.build(:strip, arguments).run!
 else 
   puts "Request-log-analyzer, by Willem van Bergen and Bart ten Brinke - version #{RequestLogAnalyzer::VERSION}"  
-  puts "Website: http://github.com/wvanbergen/request-log-analyzer"
+  puts "Website: http://railsdoctors.com"
   puts
 
   # Run the request_log_analyzer!

data/lib/cli/tools.rb

@@ -31,9 +31,9 @@ def install_rake_tasks(install_type = :rails)
   if install_type.to_sym == :rails
     require 'ftools'
     if File.directory?('./lib/tasks/')
-      File.copy(File.dirname(__FILE__) + '/../tasks/request_log_analyzer.rake', './lib/tasks/request_log_analyze.rake')
+      File.copy(File.dirname(__FILE__) + '/../../tasks/request_log_analyzer.rake', './lib/tasks/request_log_analyze.rake')
       puts "Installed rake tasks."
-      puts "To use, run: rake log:analyze"
+      puts "To use, run: rake rla:report"
     else
       puts "Cannot find /lib/tasks folder. Are you in your Rails directory?"
       puts "Installation aborted."

data/lib/request_log_analyzer/aggregator/summarizer.rb

@@ -5,23 +5,31 @@ module RequestLogAnalyzer::Aggregator
     class Definer
       
       attr_reader :trackers
-      
+
+      # Initialize tracker array
       def initialize
         @trackers = []
       end
 
+      # Initialize tracker summarizer by duping the trackers of another summarizer
+      # <tt>other</tt> The other Summarizer
       def initialize_copy(other)
         @trackers = other.trackers.dup
       end
       
+      # Drop all trackers
       def reset!
         @trackers = []
       end
       
+      # Include missing trackers through method missing.
       def method_missing(tracker_method, *args)
         track(tracker_method, args.first)
       end
       
+      # Track the frequency of a specific category
+      # <tt>category_field</tt> Field to track
+      # <tt>options</tt> options are passed to new frequency tracker
       def frequency(category_field, options = {})
         if category_field.kind_of?(Symbol)
           track(:frequency, options.merge(:category => category_field))
@@ -30,6 +38,9 @@ module RequestLogAnalyzer::Aggregator
         end
       end
       
+      # Track the duration of a specific category
+      # <tt>duration_field</tt> Field to track
+      # <tt>options</tt> options are passed to new frequency tracker
       def duration(duration_field, options = {})
         if duration_field.kind_of?(Symbol)
           track(:duration, options.merge(:duration => duration_field))
@@ -38,6 +49,9 @@ module RequestLogAnalyzer::Aggregator
         end
       end      
       
+      # Helper function to initialize a tracker and add it to the tracker array.
+      # <tt>tracker_class</tt> The class to include
+      # <tt>optiont</tt> The options to pass to the trackers.
       def track(tracker_klass, options = {})
         tracker_klass = RequestLogAnalyzer::Tracker.const_get(RequestLogAnalyzer::to_camelcase(tracker_klass)) if tracker_klass.kind_of?(Symbol)
         @trackers << tracker_klass.new(options)
@@ -47,6 +61,8 @@ module RequestLogAnalyzer::Aggregator
     attr_reader :trackers
     attr_reader :warnings_encountered
     
+    # Initialize summarizer.
+    # Generate trackers from speciefied source.file_format.report_trackers and set them up
     def initialize(source, options = {})
       super(source, options)
       @warnings_encountered = {}
@@ -57,21 +73,44 @@ module RequestLogAnalyzer::Aggregator
     def setup
     end
     
+    # Call prepare on all trackers.
     def prepare
       raise "No trackers set up in Summarizer!" if @trackers.nil? || @trackers.empty?
       @trackers.each { |tracker| tracker.prepare }
     end
     
+    # Pass all requests to trackers and let them update if necessary.
+    # <tt>request</tt> The request to pass.
     def aggregate(request)
       @trackers.each do |tracker|
         tracker.update(request) if tracker.should_update?(request)
       end
     end
     
+    # Call finalize on all trackers. Saves a YAML dump if this is set in  the options.
     def finalize
       @trackers.each { |tracker| tracker.finalize }
+      save_results_dump(options[:dump]) if options[:dump]
+    end
+    
+    # Saves the results of all the trackers in YAML format to a file.
+    # <tt>filename</tt> The file to store the YAML dump in.
+    def save_results_dump(filename)
+      File.open(filename, 'w') { |file| file.write(to_yaml) }
+    end
+    
+    # Exports all the tracker results to YAML. It will call the to_yaml_object method
+    # for every tracker and combines these into a single YAML export.
+    def to_yaml
+      require 'yaml'
+      trackers_export = @trackers.inject({}) do |export, tracker|
+        export[tracker.title] = tracker.to_yaml_object; export
+      end
+      YAML::dump(trackers_export)
     end
        
+    # Call report on all trackers.
+    # <tt>output</tt> RequestLogAnalyzer::Output object to output to
     def report(output)
       report_header(output)
       if source.parsed_requests > 0
@@ -83,6 +122,8 @@ module RequestLogAnalyzer::Aggregator
       report_footer(output)
     end
     
+    # Generate report header.
+    # <tt>output</tt> RequestLogAnalyzer::Output object to output to
     def report_header(output)
       output.title("Request summary")
   
@@ -98,6 +139,8 @@ module RequestLogAnalyzer::Aggregator
       output << "\n"
     end
     
+    # Generate report footer.
+    # <tt>output</tt> RequestLogAnalyzer::Output object to output to
     def report_footer(output)
       if has_log_ordering_warnings?
         output.title("Parse warnings")
@@ -110,14 +153,20 @@ module RequestLogAnalyzer::Aggregator
       end
     end
     
+    # Returns true if there were any warnings generated by the trackers
     def has_warnings?
-       @warnings_encountered.inject(0) { |result, (key, value)| result += value } > 0
+      @warnings_encountered.inject(0) { |result, (key, value)| result += value } > 0
     end
     
+    # Returns true if there were any log ordering warnings
     def has_log_ordering_warnings?
       @warnings_encountered[:no_current_request] && @warnings_encountered[:no_current_request] > 0
     end
     
+    # Store an encountered warning
+    # <tt>type</tt> Type of warning
+    # <tt>message</tt> Warning message
+    # <tt>lineno</tt> The line on which the error was encountered
     def warning(type, message, lineno)
       @warnings_encountered[type] ||= 0
       @warnings_encountered[type] += 1

data/lib/request_log_analyzer/controller.rb

@@ -34,6 +34,7 @@ module RequestLogAnalyzer
 
       options[:database] = arguments[:database] if arguments[:database]
       options[:debug]    = arguments[:debug]
+      options[:dump]     = arguments[:dump]
 
       output_class = RequestLogAnalyzer::Output::const_get(arguments[:output])
       if arguments[:file]
@@ -63,6 +64,7 @@ module RequestLogAnalyzer
       end
       
       controller = Controller.new(RequestLogAnalyzer::Source::LogParser.new(file_format, options), options)
+      #controller = Controller.new(RequestLogAnalyzer::Source::Database.new(file_format, options), options)
 
       options[:parse_strategy] = arguments[:parse_strategy]
       
@@ -210,6 +212,8 @@ module RequestLogAnalyzer
       if @output.io.kind_of?(File)
         puts
         puts "Report written to: " + File.expand_path(@output.io.path)
+        puts "Need an expert to analyze your application?"
+        puts "Mail to contact@railsdoctors.com or visit us at http://railsdoctors.com"
         puts "Thanks for using request-log-analyzer!"
         @output.io.close
       end

data/lib/request_log_analyzer/output/fixed_width.rb

@@ -5,17 +5,28 @@ module RequestLogAnalyzer::Output
   # Outputs a fixed width ASCII or UF8 report.
   class FixedWidth < Base
   
+    # Mixin module. Will disable any colorizing.
     module Monochrome
       def colorize(text, *options)
         text
       end
     end
   
+    # Colorize module
     module Color
     
       STYLES = { :normal => 0, :bold => 1, :underscore => 4, :blink => 5, :inverse => 7, :concealed => 8 }
       COLORS = { :black  => 0, :blue => 4, :green => 2, :cyan => 6, :red => 1, :purple => 5, :brown => 3, :white => 7 }
     
+      # Colorize text
+      # <tt>text</tt> The text to colorize
+      # Options
+      #  * <tt>:background</tt> The background color to paint. Defined in Color::COLORS
+      #  * <tt>:color</tt> The foreground color to paint. Defined in Color::COLORS
+      #  * <tt>:on</tt> Alias for :background
+      #  * <tt>:style</tt> Font style, defined in Color::STYLES
+      # 
+      # Returns ASCII colored string
       def colorize(text, *options)
     
         font_style       = ''
@@ -49,6 +60,12 @@ module RequestLogAnalyzer::Output
       :utf   => { :horizontal_line => '━', :vertical_line => '┃', :block => '░' }
     }
   
+    # Initialize a report
+    # <tt>io</tt> iO Object (file, STDOUT, etc.)
+    # <tt>options</tt>
+    #  * <tt>:characters</tt> :utf for UTF8 or :ascii for ANSI compatible output. Defaults to :utf.
+    #  * <tt>:color</tt> If true, ASCII colorization is used, else Monochrome. Defaults to Monochrome.
+    #  * <tt>:width</tt> Output width in characters. Defaults to 80.
     def initialize(io, options = {})
       super(io, options)
       @options[:width]      ||= 80
@@ -59,26 +76,36 @@ module RequestLogAnalyzer::Output
       (class << self; self; end).send(:include, color_module) 
     end
   
+    # Write a string to the output object.
+    # <tt>str</tt> The string to write.
     def print(str)
       @io << str
     end
   
     alias :<< :print
   
+    # Write a string to the output object with a newline at the end.
+    # <tt>str</tt> The string to write.
     def puts(str = '')
       @io << str << "\n"
     end
   
+    # Write the title of a report
+    # <tt>title</tt> The title to write
     def title(title)
       puts
       puts colorize(title, :bold, :white)
       line(:green)
     end
     
+    # Write a line
     def line(*font)  
       puts colorize(characters[:horizontal_line] * @options[:width], *font)
     end
-  
+
+    # Write a link
+    # <tt>text</tt> The text in the link
+    # <tt>url</tt> The url to link to.
     def link(text, url = nil)
       if url.nil?
         colorize(text, :blue, :bold)
@@ -87,6 +114,7 @@ module RequestLogAnalyzer::Output
       end
     end
     
+    # Generate a header for a report    
     def header
       if io.kind_of?(File)
         puts "Request-log-analyzer summary report"
@@ -96,11 +124,17 @@ module RequestLogAnalyzer::Output
       end
     end
     
+    # Generate a footer for a report
     def footer
       puts
+      puts "Need an expert to analyze your application?"
+      puts "Mail to contact@railsdoctors.com or visit us at http://railsdoctors.com"
       puts "Thanks for using request-log-analyzer!"
     end
     
+    # Generate a report table and push it into the output object.
+    # <tt>*colums<tt> Columns hash
+    # <tt>&block</tt>: A block yeilding the rows.
     def table(*columns, &block)
     
       rows = Array.new
@@ -150,6 +184,7 @@ module RequestLogAnalyzer::Output
         line(:green)
       end
     
+      # Print the rows
       rows.each do |row|
         row_values = []
         columns.each_with_index do |column, index|

data/lib/request_log_analyzer/output/html.rb

@@ -7,29 +7,40 @@ module RequestLogAnalyzer::Output
     #   super(io, options)
     # end
   
+    # Print a string to the io object.
     def print(str)
       @io << str
     end
 
     alias :<< :print
 
+    # Put a string with newline
     def puts(str = '')
       @io << str << "<br/>\n"
     end
 
+    # Place a title
     def title(title)
       @io.puts(tag(:h2, title))
     end
 
+    # Render a single line
+    # <tt>*font</tt> The font.
     def line(*font)  
       @io.puts(tag(:hr))
     end
-
+    
+    # Write a link
+    # <tt>text</tt> The text in the link
+    # <tt>url</tt> The url to link to.
     def link(text, url = nil)
       url = text if url.nil?
       tag(:a, text, :href => url)
     end
 
+    # Generate a report table in HTML and push it into the output object.
+    # <tt>*colums<tt> Columns hash
+    # <tt>&block</tt>: A block yeilding the rows.
     def table(*columns, &block)
       rows = Array.new
       yield(rows)
@@ -56,6 +67,7 @@ module RequestLogAnalyzer::Output
     
     end
   
+    # Genrate HTML header and associated stylesheet
     def header
       @io << "<html>"
       @io << tag(:head) do |headers|
@@ -127,14 +139,20 @@ module RequestLogAnalyzer::Output
       @io << tag(:p, "Version #{RequestLogAnalyzer::VERSION} - written by Willem van Bergen and Bart ten Brinke")
     end
   
+    # Generate a footer for a report
     def footer
       @io << tag(:hr) << tag(:h2, 'Thanks for using request-log-analyzer')
-      @io << tag(:p, 'Please visit the ' + link('Request-log-analyzer website', 'http://github.com/wvanbergen/request-log-analyzer'))
+      @io << tag(:p, 'For more information please visit the ' + link('Request-log-analyzer website', 'http://github.com/wvanbergen/request-log-analyzer'))
+      @io << tag(:p, 'If you need an expert who can analyze your application, mail to ' + link('contact@railsdoctors.com', 'mailto:contact@railsdoctors.com') + ' or visit us at ' + link('http://railsdoctors.com', 'http://railsdoctors.com') + '.')
       @io << "</body></html>\n"
     end
   
     protected 
   
+    # HTML tag writer helper
+    # <tt>tag</tt> The tag to generate
+    # <tt>content</tt> The content inside the tag
+    # <tt>attributes</tt> Attributes to write in the tag
     def tag(tag, content = nil, attributes = nil)
       if block_given?
         attributes = content.nil? ? '' : ' ' + content.map { |(key, value)| "#{key}=\"#{value}\"" }.join(' ')

data/lib/request_log_analyzer/output.rb

@@ -1,19 +1,27 @@
+# Module for generating output
 module RequestLogAnalyzer::Output
 
+  # Load class files if needed
   def self.const_missing(const)
     RequestLogAnalyzer::load_default_class_file(self, const)
   end
   
+  # Base Class used for generating output for reports.
+  # All output should inherit fromt this class.
   class Base
     
     attr_accessor :io, :options, :style
     
+    # Initialize a report
+    # <tt>io</tt> iO Object (file, STDOUT, etc.)
+    # <tt>options</tt> Specific style options
     def initialize(io, options = {})
       @io      = io
       @options = options
       @style   = options[:style] || { :cell_separator => true, :table_border => false }
     end
 
+    # Apply a style block.. with style :)
     def with_style(temp_style = {})
       old_style = @style
       @style = @style.merge(temp_style)
@@ -21,14 +29,41 @@ module RequestLogAnalyzer::Output
       @style = old_style
     end    
     
+    # Generate a header for a report
     def header
     end
     
+    # Generate the footer of a report 
     def footer
     end
+
+    # Generate a report table and push it into the output object.
+    # Yeilds a rows array into which the rows can be pushed
+    # <tt>*colums<tt> Array of Column hashes (see Column options).
+    # <tt>&block</tt>: A block yeilding the rows.
+    # 
+    # === Column options
+    # Columns is an array of hashes containing the column definitions.
+    # * <tt>:align</tt> Alignment :left or :right
+    # * <tt>:treshold</tt> Width in characters or :rest
+    # * <tt>:type</tt> :ratio or nil
+    # * <tt>:width</tt> Width in characters or :rest
+    # 
+    # === Example
+    # The output object should support table definitions:
+    #
+    # output.table({:align => :left}, {:align => :right }, {:align => :right}, {:type => :ratio, :width => :rest}) do |rows|        
+    #   sorted_frequencies.each do |(cat, count)|
+    #     rows << [cat, "#{count} hits", '%0.1f%%' % ((count.to_f / total_hits.to_f) * 100.0), (count.to_f / total_hits.to_f)]
+    #   end
+    # end
+    #
+    def table(*columns, &block)
+    end
     
     protected
-    
+    # Check if a given table defination hash includes a header (title)
+    # <tt>columns</tt> The columns hash
     def table_has_header?(columns)
       columns.any? { |column| !column[:title].nil? } 
     end

data/lib/request_log_analyzer/request.rb

@@ -60,9 +60,9 @@ module RequestLogAnalyzer
         
     # Initializes a new Request object. 
     # It will apply the the provided FileFormat module to this instance.
-    def initialize(file_format)
+    def initialize(file_format, attributes = {})
       @lines = []
-      @attributes = {}
+      @attributes = attributes
       register_file_format(file_format)
     end
     

data/lib/request_log_analyzer/source/database.rb

@@ -4,12 +4,25 @@ require 'activerecord'
 module RequestLogAnalyzer::Source
   
   # Active Resource hook
-  class CompletedLine < ActiveRecord::Base
+  class Request < ActiveRecord::Base
+    has_many :completed_lines
+    has_many :processing_lines
     def convert(file_format)
-      RequestLogAnalyzer::Request.create(file_format, self.attributes)
+      send_attributes = self.attributes
+      send_attributes.merge!(self.completed_lines.first.attributes) if self.completed_lines.first
+      send_attributes.merge!(self.processing_lines.first.attributes) if self.processing_lines.first
+      return RequestLogAnalyzer::Request.new(file_format, send_attributes)
     end
   end
 
+  class CompletedLine < ActiveRecord::Base
+    belongs_to :request
+  end
+
+  class ProcessingLine < ActiveRecord::Base
+    belongs_to :request
+  end
+
   # The Database class gets log data from the database.
   class Database < Base
 
@@ -40,9 +53,11 @@ module RequestLogAnalyzer::Source
       ActiveRecord::Base.establish_connection(:adapter => 'sqlite3', :database => @source_files)
 
       @progress_handler.call(:started, @source_files) if @progress_handler
-        RequestLogAnalyzer::Source::CompletedLine.find(:all).each do |request|
+        RequestLogAnalyzer::Source::Request.find(:all).each do |request|
           @parsed_requests += 1
-          yield(request.convert(self.file_format))
+          @progress_handler.call(:progress, @parsed_requests) if @progress_handler
+          
+          yield request.convert(self.file_format)
         end
       
       @progress_handler.call(:finished, @source_files) if @progress_handler