production_log_analyzer 1.1.0

data/LICENSE

@@ -0,0 +1,27 @@
+Copyright 2005 Eric Hodel, The Robot Co-op.  All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions
+are met:
+
+1. Redistributions of source code must retain the above copyright
+   notice, this list of conditions and the following disclaimer.
+2. Redistributions in binary form must reproduce the above copyright
+   notice, this list of conditions and the following disclaimer in the
+   documentation and/or other materials provided with the distribution.
+3. Neither the names of the authors nor the names of their contributors
+   may be used to endorse or promote products derived from this software
+   without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE AUTHORS ``AS IS'' AND ANY EXPRESS
+OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE
+LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,
+OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
+OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
+WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
+OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
+EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+

data/Manifest.txt

@@ -0,0 +1,11 @@
+Rakefile
+README
+LICENSE
+Manifest.txt
+bin/pl_analyze
+lib/production_log/analyzer.rb
+lib/production_log/parser.rb
+lib/production_log/syslog_logger.rb
+test/test.syslog.log
+test/test_analyzer.rb
+test/test_parser.rb

data/README

@@ -0,0 +1,121 @@
+Production Log Analyzer
+
+http://rails-analyzer.rubyforge.org
+
+http://rubyforge.org/projects/rails-analyzer
+
+= About
+
+The Production Analyzer lets you find out which pages on your site are
+dragging you down.  PL Analyze requires the use of SyslogLogger (included)
+because the default Logger doesn't give any way to associate lines logged to a
+request.
+
+SyslogLogger gives you many other advantages, such as the ability to combine
+logs from multiple machines.
+
+= Installing
+
+== Download
+
+Grab the gem (or tar.gz or .zip) from the RubyForge project:
+
+http://rubyforge.org/frs/?group_id=586
+
+== What you need
+
+Either:
+
+A syslogd that doesn't suck.  This means that +man syslog.conf+ shows a
+!prog specification.  (FreeBSD's syslogd doesn't suck, but OS X's syslogd
+does.)
+
+or:
+
+Some hacking skills to make Rails log the pid of the process for each line
+logged.  You'll also have to teach LogParser#parse about this.  Feel free to
+submit patches with tests.  (Patches without tests are useless to me.)
+
+== What you need to do
+
+Either:
+
+Use SyslogLogger according to directions on the SyslogLogger page, including
+setting up your non-sucky syslogd as directed.
+
+or:
+
+Use your hacking skills and set up a logger that your hacked LogParser#parse
+can deal with.
+
+Then:
+
+Set up a cronjob (or something like that) to run log files through pl_analyze.
+
+= Using pl_analyze
+
+To run pl_analyze simply give it the name of a log file to analyze.
+
+  pl_analyze /var/log/production.log
+
+If you want, you can run it from a cron something like this:
+
+  /usr/bin/gzip -dc /var/log/production.log.0.gz | /usr/local/bin/pl_analyze /dev/stdin
+
+In the future, pl_analyze will be able to read from STDIN.
+
+= Sample output
+
+  Average Request Time: 0.279874593327209
+  Request Time Std Dev: 0.351590385021209
+
+  Slowest Request Times:
+          ZeitgeistController#goals took 30.889858s
+          ZeitgeistController#goals took 29.657513s
+          EntriesController#save_comment took 20.499292s
+          AccountController#create took 19.539545s
+          EntriesController#save_comment took 15.46844s
+          ZeitgeistController#goals took 14.814086s
+          ZeitgeistController#goals took 13.943129s
+          ZeitgeistController#goals took 13.113908s
+          ZeitgeistController#completed_goals took 12.776777s
+          ZeitgeistController#goals took 12.32529s
+
+  Average DB Time: 0.0649204642242509
+  DB Time Std Dev: 0.214050667483775
+
+  Slowest Total DB Times:
+          ZeitgeistController#goals took 30.797014s
+          ZeitgeistController#goals took 29.567076s
+          ZeitgeistController#goals took 14.709733s
+          ZeitgeistController#goals took 13.84484s
+          ZeitgeistController#goals took 12.968071s
+          ZeitgeistController#completed_goals took 12.400506s
+          ZeitgeistController#goals took 12.241167s
+          ZeitgeistController#goals took 11.561719s
+          ZeitgeistController#goals took 11.445382s
+          ZeitgeistController#goals took 11.085795s
+
+  Average Render Time: 0.128757978789508
+  Render Time Std Dev: 0.131171213785894
+
+  Slowest Total Render Times:
+          TeamsController#progress took 4.698406s
+          TeamsController#progress took 4.679505s
+          PeopleController#doing_same_things took 3.628557s
+          ThingsController#view took 3.34039s
+          ThingsController#view took 2.096405s
+          RssController#goals took 1.759452s
+          EntriesController#view took 1.423261s
+          ThingsController#view took 1.422453s
+          ThingsController#people took 1.377157s
+          PeopleController#view took 1.195831s
+
+= What's missing
+
+* More reports
+* Command line arguments including:
+  * Number of lines to report
+  * What type of log file you've got (if somebody sends patches with tests)
+* Lots more
+

data/Rakefile

@@ -0,0 +1,62 @@
+require 'rubygems'
+require 'rake'
+require 'rake/testtask'
+require 'rake/rdoctask'
+require 'rake/gempackagetask'
+require 'rake/contrib/sshpublisher'
+
+$VERBOSE = nil
+
+spec = Gem::Specification.new do |s|
+    s.name = "production_log_analyzer"
+    s.version = "1.1.0"
+    s.summary = "Extracts statistics from Rails production logs"
+    s.author = "Eric Hodel"
+    s.email = "hodel@robotcoop.com"
+
+    s.has_rdoc = true
+    s.files = File.read("Manifest.txt").split($/)
+    s.require_path = 'lib'
+    s.executables = ["pl_analyze"]
+    s.default_executable = "pl_analyze"
+end
+
+desc "Run tests"
+task :default => [ :test ]
+
+Rake::TestTask.new("test") do |t|
+    t.libs << "test"
+    t.pattern = "test/test_*.rb"
+    t.verbose = true
+end
+
+desc "Generate RDoc"
+Rake::RDocTask.new :rdoc do |rd|
+    rd.rdoc_dir = "doc"
+    rd.rdoc_files.add "lib", "README", "LICENSE"
+    rd.main = "README"
+    rd.options << "-d" if `which dot` =~ /\/dot/
+end
+
+desc "Build Gem"
+Rake::GemPackageTask.new spec do |pkg|
+    pkg.need_zip = false
+    pkg.need_tar = false
+end
+
+desc "Sends RDoc to RubyForge"
+task :send_rdoc => [ :rerdoc ] do
+    publisher = Rake::SshDirPublisher.new('drbrain@rubyforge.org',
+                    '/var/www/gforge-projects/rails-analyzer',
+                    'doc')
+    publisher.upload
+end
+
+desc "Clean up"
+task :clean => [ :clobber_rdoc, :clobber_package ]
+
+desc "Clean up"
+task :clobber => [ :clean ]
+
+# vim: ts=4 sts=4 sw=4 syntax=Ruby
+

data/bin/pl_analyze

@@ -0,0 +1,50 @@
+#!/usr/local/bin/ruby -w
+
+require 'production_log/analyzer'
+
+file_name = ARGV.shift
+
+if file_name.nil? then
+  puts "Usage: #{$0} file_name"
+  exit 1
+end
+
+analyzer = Analyzer.new file_name
+analyzer.process
+
+count = ARGV.shift
+count = count.nil? ? 10 : count.to_i
+
+request_times = analyzer.slowest_request_times(count).map do |time, name|
+  "\t#{name} took #{time}s"
+end
+
+puts "Average Request Time: #{analyzer.average_request_time}"
+puts "Request Time Std Dev: #{analyzer.request_time_std_dev}"
+puts
+puts "Slowest Request Times:"
+puts request_times.join($/)
+puts
+
+db_times = analyzer.slowest_db_times(count).map do |time, name|
+  "\t#{name} took #{time}s"
+end
+
+puts "Average DB Time: #{analyzer.average_db_time}"
+puts "DB Time Std Dev: #{analyzer.db_time_std_dev}"
+puts
+puts "Slowest Total DB Times:"
+puts db_times.join($/)
+puts
+
+render_times = analyzer.slowest_render_times(count).map do |time, name|
+  "\t#{name} took #{time}s"
+end
+
+puts "Average Render Time: #{analyzer.average_render_time}"
+puts "Render Time Std Dev: #{analyzer.render_time_std_dev}"
+puts
+puts "Slowest Total Render Times:"
+puts render_times.join($/)
+puts
+

data/lib/production_log/analyzer.rb

@@ -0,0 +1,266 @@
+#!/usr/local/bin/ruby -w
+
+require 'production_log/parser'
+
+module Enumerable
+
+    ##
+    # Sum of all the elements of the Enumerable
+
+    def sum
+        return self.inject(0) { |acc, i| acc + i }
+    end
+
+    ##
+    # Average of all the elements of the Enumerable
+    #
+    # The Enumerable must respond to #length
+
+    def average
+        return self.sum / self.length.to_f
+    end
+
+    ##
+    # Sample variance of all the elements of the Enumerable
+    #
+    # The Enumerable must respond to #length
+
+    def sample_variance
+        avg = self.average
+        sum = self.inject(0) { |acc, i| acc + (i - avg) ** 2 }
+        return (1 / self.length.to_f * sum)
+    end
+
+    ##
+    # Standard deviation of all the elements of the Enumerable
+    #
+    # The Enumerable must respond to #length
+
+    def standard_deviation
+        return Math.sqrt(self.sample_variance)
+    end
+
+end
+
+##
+# A list that only stores +limit+ items.
+
+class SizedList < Array
+
+    ##
+    # Creates a new SizedList that can hold up to +limit+ items.  Whenever
+    # adding a new item to the SizedList would make the list larger than
+    # +limit+, +delete_block+ is called.
+    #
+    # +delete_block+ is passed the list and the item being added.
+    # +delete_block+ must take action to remove an item and return true or
+    # return false if the item should not be added to the list.
+
+    def initialize(limit, &delete_block)
+        @limit = limit
+        @delete_block = delete_block
+    end
+
+    ##
+    # Attempts to add +obj+ to the list.
+
+    def <<(obj)
+        return super if self.length < @limit
+        return super if @delete_block.call self, obj
+    end
+
+end
+
+##
+# Stores +limit+ time/object pairs, keeping only the largest +limit+ items.
+#
+# Sample usage:
+#
+#   l = SlowestTimes.new 5
+#   
+#   l << [Time.now, 'one']
+#   l << [Time.now, 'two']
+#   l << [Time.now, 'three']
+#   l << [Time.now, 'four']
+#   l << [Time.now, 'five']
+#   l << [Time.now, 'six']
+#
+#   p l.map { |i| i.last }
+
+class SlowestTimes < SizedList
+
+    ##
+    # Creates a new SlowestTimes SizedList that holds only +limit+ time/object
+    # pairs.
+    def initialize(limit)
+        super limit do |arr, new_item|
+            fastest_time = arr.sort_by { |time, name| time }.first
+            if fastest_time.first < new_item.first then
+                arr.delete_at index(fastest_time)
+                true
+            else
+                false
+            end
+        end
+    end
+
+end
+
+##
+# Calculates statistics for production logs.
+
+class Analyzer
+
+    ##
+    # The logfile being read by the Analyzer.
+
+    attr_reader :logfile_name
+
+    ##
+    # An Array of all the request total times for the log file.
+
+    attr_reader :request_times
+
+    ##
+    # An Array of all the request database times for the log file.
+
+    attr_reader :db_times
+
+    ##
+    # An Array of all the request render times for the log file.
+
+    attr_reader :render_times
+
+#    attr_reader :query_times
+
+    ##
+    # Creates a new Analyzer that will read data from +logfile_name+.
+
+    def initialize(logfile_name)
+        @logfile_name  = logfile_name
+        @request_times = Hash.new { |h,k| h[k] = [] }
+        @db_times      = Hash.new { |h,k| h[k] = [] }
+        @render_times  = Hash.new { |h,k| h[k] = [] }
+#        @query_times   = Hash.new { |h,k| h[k] = [] }
+    end
+
+    ##
+    # Processes the log file collecting statistics from each found LogEntry.
+
+    def process
+        File.open @logfile_name do |fp|
+            LogParser.parse fp do |entry|
+                entry_page = entry.page
+                @request_times[entry_page] << entry.request_time
+                @db_times[entry_page] << entry.db_time
+                @render_times[entry_page] << entry.render_time
+#                entry.queries.each do |name, time, sql|
+#                    @query_times[entry_page] << time 
+#                end
+            end
+        end
+    end
+
+    ##
+    # The average total request time for all requests.
+
+    def average_request_time
+        times = @request_times.values.flatten
+        times.delete 0
+        return times.average
+    end
+
+    ##
+    # The standard deviation of the total request time for all requests.
+
+    def request_time_std_dev
+        times = @request_times.values.flatten
+        times.delete 0
+        return times.standard_deviation
+    end
+
+    ##
+    # The +limit+ slowest total request times.
+
+    def slowest_request_times(limit = 10)
+        slowest_times = SlowestTimes.new limit
+
+        @request_times.each do |name, times|
+            times.each do |time|
+                slowest_times << [time, name]
+            end
+        end
+
+        return slowest_times.sort_by { |time, name| time }.reverse
+    end
+
+    ##
+    # The average total database time for all requests.
+
+    def average_db_time
+        times = @db_times.values.flatten
+        times.delete 0
+        return times.average
+    end
+
+    ##
+    # The standard deviation of the total database time for all requests.
+
+    def db_time_std_dev
+        times = @db_times.values.flatten
+        times.delete 0
+        return times.standard_deviation
+    end
+
+    ##
+    # The +limit+ slowest total database times.
+
+    def slowest_db_times(limit = 10)
+        slowest_times = SlowestTimes.new limit
+
+        @db_times.each do |name, times|
+            times.each do |time|
+                slowest_times << [time, name]
+            end
+        end
+
+        return slowest_times.sort_by { |time, name| time }.reverse
+    end
+
+    ##
+    # The average total render time for all requests.
+
+    def average_render_time
+        times = @render_times.values.flatten
+        times.delete 0
+        return times.average
+    end
+
+    ##
+    # The standard deviation of the total render time for all requests.
+
+    def render_time_std_dev
+        times = @render_times.values.flatten
+        times.delete 0
+        return times.standard_deviation
+    end
+
+    ##
+    # The +limit+ slowest total render times for all requests.
+
+    def slowest_render_times(limit = 10)
+        slowest_times = SlowestTimes.new limit
+
+        @render_times.each do |name, times|
+            times.each do |time|
+                slowest_times << [time, name]
+            end
+        end
+
+        return slowest_times.sort_by { |time, name| time }.reverse
+    end
+
+end
+
+# vim: ts=4 sts=4 sw=4
+