Narnach-rails_analyzer 0.2.2

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2008 Wes Oldenbeuving
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.rdoc

@@ -0,0 +1,58 @@
+== RailsAnalyzer
+RailsAnalyzer generates reports about requests processed by a Ruby on Rails server.
+
+It analyzes the log files created by a Rails server. It can accept any number of files as its command line parameters. It defaults to use 'log/production.log' as its input when no filenames are provided.
+
+Two different reporting engines process the log files: 
+* TimeStats, which is concerned with *when* the request was made. It produces reports with the amount of hits recorded in the log file per time interval, and the relative amount of requests within a time interval. The relative interval simply lowers all hit counts until one hits zero. This makes it easy to filter out periodic requests.
+* HitStats, which is concerned with how *fast* the requests were handled. It produces reports with detailed time statistics for each request. It has two types of reports: with params and without params. Each type of report produces a number of files, each which is sorted by one of the following statistics for responses: fastest, slowest, sum, average, median, standard deviation and hit count.
+Both engines produce .txt files as their only output.
+* TimeStats produces log_times_*.txt and log_times_*_relative.txt output.
+* HitStats produces the other log_*.txt and log_*_with_params.txt files.
+
+== Recent changes
+
+=== Version 0.2.2
+Added reports for slowest and fastest requests.
+Documentation updates.
+
+=== Version 0.2.1
+Bugfixes:
+* HitStats reports were not generated. They are working again.
+* Array#median always returns a float to prevent integer math (lack of) rounding problems.
+Dependencies:
+* No longer rely on ActiveSupport. It was used for Array#sum and Array#group_by, which are now implemented in ArrayExt.
+Specs:
+* Parts of UrlHits and Entries got specs to help debugging the HitStats bug.
+* Added specs for all ArrayExt methods.
+
+=== Version 0.2.0
+Split single-file script into one file per existing class.
+Introduced new classes to handle responsibilities that were not yet handled.
+Changed classes involved in generating URL hits-based reports to be more flexible.
+Generally refactored a lot of non-DRY code to be at least a bit nicer.
+
+=== Version 0.1.0
+Imported single-file script
+
+== Installation
+=== From gem
+The gem is located on github.
+  gem install Narnach-rails_analyzer -s http://gems.github.com
+=== From git
+From the project root, use rake to install:
+  git clone git://github.com/Narnach/rails_analyzer.git
+  cd rails_analyzer
+  rake install
+This will build the gem and install it for you.
+
+== Syntax
+  rails_analyzer [log_file1] [log_file2] [..] [log_fileN]
+When no log files are provided, log/production.log is used.
+
+== About
+
+Author:: Wes 'Narnach' Oldenbeuving (narnach@gmail.com)
+Website:: http://www.github.com/Narnach/rails_analyzer
+Copyright:: Copyright (c) 2008 Wes Oldenbeuving
+License:: MIT license. See MIT-LICENSE (in the gem directory) for license details.

data/Rakefile

@@ -0,0 +1,28 @@
+require "rake"
+require "rake/clean"
+require "rake/gempackagetask"
+require 'rubygems'
+
+################################################################################
+### Gem
+################################################################################
+
+begin
+  # Parse gemspec using the github safety level.
+  file = Dir['*.gemspec'].first
+  data = File.read(file)
+  spec = nil
+  Thread.new { spec = eval("$SAFE = 3\n%s" % data)}.join
+
+  # Create the gem tasks
+  Rake::GemPackageTask.new(spec) do |package|
+    package.gem_spec = spec
+  end
+rescue Exception => e
+  printf "WARNING: Error caught (%s): %s\n%s", e.class.name, e.message, e.backtrace[0...5].map {|l| '  %s' % l}.join("\n")
+end
+
+desc 'Package and install the gem for the current version'
+task :install => :gem do
+  system "sudo gem install -l pkg/%s-%s.gem" % [spec.name, spec.version]
+end

data/bin/rails_analyzer

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+require 'rails_analyzer'
+
+ra = RailsAnalyzer.new(ARGV)
+ra.generate_reports

data/lib/array_ext.rb

@@ -0,0 +1,51 @@
+module ArrayExt
+  module GroupBy
+    # Returns a Hash:
+    # - The keys are grouping values
+    # - The values are an Array with values grouped to that key.
+    def group_by(&block)
+      grouped_results = Hash.new { |hash, key| hash[key] = Array.new }
+      each do |element|
+        group_key = block.call(element)
+        grouped_results[group_key] << element
+      end
+      grouped_results
+    end
+  end
+
+  module Stats
+    def avg
+      sum / size.to_f
+    end
+
+    # Returns the median as a Float.
+    def median
+      return 0 if size == 0
+      if size%2==0
+        # Average two middle values
+        # [1,2,3,4,5,6].median #=> 3.5
+        (self[size / 2] + self[size / 2 - 1]) / 2.0
+      else
+        # Use middle value
+        # [1,2,3,4,5].median #=> 3
+        self[size / 2].to_f
+      end
+    end
+
+    def stddev
+      avg_cached = avg # prevent having to recompute it each time
+      squared_deviations = map {|n| (n - avg_cached) ** 2 }
+      variance = squared_deviations.avg
+      Math::sqrt(variance)
+    end
+
+    def sum
+      inject(0.0) {|s,n| s+n }
+    end
+  end
+end
+
+class Array
+  include ArrayExt::GroupBy
+  include ArrayExt::Stats
+end

data/lib/entries.rb

@@ -0,0 +1,19 @@
+require 'url_hits'
+
+class Entries
+  attr_accessor :entries
+
+  def initialize
+    @entries = Hash.new { |hash, url| hash[url] = UrlHits.new(url) }
+  end
+
+  def add_hit(url, time)
+    entries[url].add_hit(time)
+  end
+
+  def to_s(sort=:size)
+    sorted = entries.values.sort {|a,b| b.send(sort) <=> a.send(sort) }
+    head = ('%6s ' + '%6s '*5 + '%9s %s') % %w[Hits Low Median Avg Stddev High Sum Url]
+    head << "\n" << sorted.join("\n")
+  end
+end

data/lib/float_ext.rb

@@ -0,0 +1,18 @@
+# Add decent rounding to X decimals.
+module FloatExt
+  def round_to(x=4)
+    (self * 10**x).round.to_f / 10**x
+  end
+
+  def ceil_to(x=4)
+    (self * 10**x).ceil.to_f / 10**x
+  end
+
+  def floor_to(x=4)
+    (self * 10**x).floor.to_f / 10**x
+  end
+end
+
+class Float
+  include FloatExt
+end

data/lib/hit_stats.rb

@@ -0,0 +1,64 @@
+require 'entries'
+require 'uri'
+
+class HitStats
+  attr_accessor :logs, :hits_with_query, :hits_without_query
+
+  def initialize(logs)
+    @logs = logs
+    @hits_with_query = Entries.new
+    @hits_without_query = Entries.new
+  end
+
+  def self.generate(logs)
+    hs = self.new(logs)
+    hs.parse_logs
+    hs.save_reports
+  end
+
+  def parse_logs
+    logs.each do |log|
+      parse_log(log)
+    end
+  end
+
+  def save_reports
+    [:sum, :avg, :size, :median, :stddev, :min, :max].each do |sort_order|
+      File.open("log_%s.txt" % name_for_sort_order(sort_order),"wb") {|file| file.puts hits_without_query.to_s(sort_order) }
+      File.open('log_%s_with_params.txt' % name_for_sort_order(sort_order), 'wb') {|file| file.puts hits_with_query.to_s(sort_order)}
+    end
+  end
+
+  protected
+
+  def name_for_sort_order(order)
+    {
+      :size => 'hits',
+      :min => 'low',
+      :max => 'high',
+    }[order] || order
+  end
+
+  def parse_log(log)
+    results = `grep "Completed" #{log}`
+    results.each do |line|
+      parse_line(line)
+    end
+  end
+  
+  def parse_line(line)
+    unless line =~ /\[(.*?)\]/
+      puts "Failed to parse: #{line}"
+      return
+    end
+    return unless uri=URI.parse($1) rescue nil
+    unless line =~ /Completed\ in\ ([0-9]+\.[0-9]+)/
+      puts "Could not extract time from line: #{line}"
+      return
+    end
+    time = $1.to_f
+    hits_with_query.add_hit(uri.to_s,time)
+    uri.query=nil
+    hits_without_query.add_hit(uri.to_s,time)
+  end
+end

data/lib/rails_analyzer.rb

@@ -0,0 +1,23 @@
+require 'time_stats'
+require 'hit_stats'
+
+class RailsAnalyzer
+  attr_accessor :logs
+
+  def initialize(logs=[])
+    @logs = logs
+    @logs = %w[log/production.log] if logs.size==0
+  end
+
+  def generate_reports
+    puts "RailsAnalyzer"
+    puts "Analyzes the following Rails log files:"
+    puts logs.map{|l| '  %s' % l}.join("\n")
+
+    puts 'Generating time-based reports'
+    TimeStats.generate(logs)
+
+    puts 'Generating hit-based reports'
+    HitStats.generate(logs)
+  end
+end

data/lib/time_stats.rb

@@ -0,0 +1,79 @@
+require 'array_ext'
+
+class TimeStats
+  attr_accessor :logs, :times
+
+  def initialize(logs)
+    @times = []
+    @logs = logs
+  end
+
+  def self.generate(logs)
+    ts = new(logs)
+    ts.parse_files
+    ts.save_reports
+  end
+
+  def parse_files
+    logs.each do |log|
+      results = `grep "Processing" #{log}`
+      timestamps = results.map { |line| (line =~ /Processing.*\(for .* at (\d+-\d+-\d+ \d+:\d+:\d+)\)/) ? $1 : nil}.compact
+      times.concat(timestamps)
+    end
+  end
+
+  def save_reports
+    %w[day hour day_hour].each do |timeframe|
+      filename = 'log_times_%s.txt' % timeframe
+      save_hits_report(filename, timeframe)
+    end
+    %w[hour ten_min].each do |timeframe|
+      hits = self.send('per_%s' % timeframe).map {|frame, hits| hits.size}
+      offset = hits.min || 0
+      filename = 'log_times_%s_relative.txt' % timeframe
+      save_hits_report(filename, timeframe, offset)
+    end
+  end
+
+  protected
+  
+  def save_hits_report(filename, timeframe, offset = 0)
+    collection = self.send('per_%s' % timeframe)
+    File.open(filename,'wb') do |f|
+      collection.each do |frame, hits|
+        f.puts '%s: %s' % [frame, hits.size - offset]
+      end
+    end
+  end
+
+  def per_day_hour
+    group_times_by {|d, h, m, s| '%s %sh' % [d, h]}
+  end
+
+  def per_day
+    group_times_by {|d, h, m, s| d}
+  end
+
+  def per_hour
+    group_times_by {|d, h, m, s| h}
+  end
+
+  def per_ten_min
+    group_times_by {|d, h, m, s| '%sh%s0' % [h, m[0,1]]}
+  end
+
+  # Group times by date/time-related data.
+  # Sorts by the yield return value
+  # Returns an Array of two-element Arrays:
+  # - The first element is the sort key
+  # - The second element is the hit times for that key
+  # The return value is sorted by key.
+  def group_times_by(&block) # :yields: date, hour, min, sec
+    grouped_times = times.group_by do |t|
+      date, time = t.split(" ")
+      hour, min, sec = time.split(":")
+      block.call(date, hour, min, sec)
+    end
+    grouped_times.to_a.sort {|a,b| a.first <=> b.first}
+  end
+end

data/lib/url_hits.rb

@@ -0,0 +1,36 @@
+require 'float_ext'
+require 'array_ext'
+
+class UrlHits
+  attr_reader :url, :hits
+  
+  def initialize(url)
+    @url = url
+    @hits = Array.new
+  end
+
+  def add_hit(time)
+    hits << time
+  end
+
+  def to_s
+    '%6i %s %s' % [size, time_stats, @url]
+  end
+
+  def ==(other)
+    (other.class == self.class) && (other.url == self.url) && (other.hits == self.hits)
+  end
+
+  [:size, :min, :max, :sum, :stddev, :median, :avg].each do |op|
+    define_method(op) do
+      hits.send(op)
+    end
+  end
+
+  protected
+
+  def time_stats
+    return nil unless size > 0
+    '%3.03f %3.03f %3.03f %3.03f %3.03f %6.03f' % [min, median, avg, stddev, max, sum]
+  end
+end

data/rails_analyzer.gemspec

@@ -0,0 +1,33 @@
+Gem::Specification.new do |s|
+  # Project
+  s.name         = 'rails_analyzer'
+  s.summary      = "RailsAnalyzer generates reports about requests processed by a Ruby on Rails server."
+  s.description  = "RailsAnalyzer generates reports about requests processed by a Ruby on Rails server."
+  s.version      = '0.2.2'
+  s.date         = '2008-09-15'
+  s.platform     = Gem::Platform::RUBY
+  s.authors      = ["Wes Oldenbeuving"]
+  s.email        = "narnach@gmail.com"
+  s.homepage     = "http://www.github.com/Narnach/rails_analyzer"
+
+  # Files
+  root_files     = %w[MIT-LICENSE README.rdoc Rakefile rails_analyzer.gemspec]
+  bin_files      = %w[rails_analyzer]
+  lib_files      = %w[rails_analyzer array_ext float_ext time_stats hit_stats entries url_hits]
+  test_files     = %w[]
+  spec_files     = %w[rails_analyzer entries url_hits]
+  other_files    = %w[spec/spec.opts spec/spec_helper.rb]
+  s.bindir       = "bin"
+  s.require_path = "lib"
+  s.executables  = bin_files
+  s.test_files   = test_files.map {|f| 'test/%s_test.rb' % f} + spec_files.map {|f| 'spec/%s_spec.rb' % f}
+  s.files        = root_files + s.test_files + other_files + bin_files.map {|f| 'bin/%s' % f} + lib_files.map {|f| 'lib/%s.rb' % f}
+
+  # rdoc
+  s.has_rdoc         = true
+  s.extra_rdoc_files = %w[ README.rdoc MIT-LICENSE]
+  s.rdoc_options << '--inline-source' << '--line-numbers' << '--main' << 'README.rdoc'
+
+  # Requirements
+  s.required_ruby_version = ">= 1.8.0"
+end

data/spec/entries_spec.rb

@@ -0,0 +1,46 @@
+require File.dirname(__FILE__) + '/spec_helper'
+require 'entries'
+
+describe Entries do
+  describe '#add_hit' do
+    it 'should be stored grouped per URL' do
+      u = 'http://www.example.org'
+      u2 = 'http://www2.example.org'
+      uh1 = UrlHits.new(u)
+      uh1.add_hit(0.1)
+      uh2 = UrlHits.new(u)
+      uh2.add_hit(0.1)
+      uh2.add_hit(0.2)
+      uh3 = UrlHits.new(u)
+      uh3.add_hit(0.1)
+      uh3.add_hit(0.2)
+      uh3.add_hit(0.3)
+      uh4 = UrlHits.new(u2)
+      uh4.add_hit(0.1)
+
+      e = Entries.new
+      e.entries.should == {}
+      e.add_hit(u, 0.1)
+      e.entries.should == {u => uh1}
+      e.add_hit(u, 0.2)
+      e.entries.should == {u => uh2}
+      e.add_hit(u, 0.3)
+      e.entries.should == {u => uh3}
+      e.add_hit(u2, 0.1)
+      e.entries.should == {u => uh3, u2 => uh4}
+    end
+  end
+
+  describe '#to_s' do
+    it "should return a summary of the url hit times" do
+      e = Entries.new
+      e.add_hit('http://www.example.org', 0.1)
+      e.add_hit('http://www.example.org', 0.2)
+      e.add_hit('http://www.example.org', 0.3)
+      expected_output = '' + \
+        '  Hits    Low Median    Avg Stddev   High       Sum Url' + "\n" + \
+        '     3 0.100 0.200 0.200 0.082 0.300  0.600 http://www.example.org'
+      e.to_s.should == expected_output
+    end
+  end
+end

data/spec/rails_analyzer_spec.rb

@@ -0,0 +1,5 @@
+require File.dirname(__FILE__) + '/spec_helper'
+require 'rails_analyzer'
+
+describe RailsAnalyzer do
+end

data/spec/spec.opts

@@ -0,0 +1,3 @@
+--colour
+--format specdoc
+--diff unified

data/spec/spec_helper.rb

@@ -0,0 +1 @@
+$LOAD_PATH.unshift File.expand_path(File.join(File.dirname(__FILE__),'..','lib'))

data/spec/url_hits_spec.rb

@@ -0,0 +1,16 @@
+require File.dirname(__FILE__) + '/spec_helper'
+require 'url_hits'
+
+describe UrlHits do
+  describe '#to_s' do
+    it "should return a summary of the url hit times" do
+      uh = UrlHits.new('http://www.example.org')
+      uh.add_hit(0.1)
+      uh.to_s.should == '     1 0.100 0.100 0.100 0.000 0.100  0.100 http://www.example.org'
+      uh.add_hit(0.2)
+      uh.to_s.should == '     2 0.100 0.150 0.150 0.050 0.200  0.300 http://www.example.org'
+      uh.add_hit(0.3)
+      uh.to_s.should == '     3 0.100 0.200 0.200 0.082 0.300  0.600 http://www.example.org'
+    end
+  end
+end

metadata

@@ -0,0 +1,75 @@
+--- !ruby/object:Gem::Specification 
+name: Narnach-rails_analyzer
+version: !ruby/object:Gem::Version 
+  version: 0.2.2
+platform: ruby
+authors: 
+- Wes Oldenbeuving
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2008-09-15 00:00:00 -07:00
+default_executable: 
+dependencies: []
+
+description: RailsAnalyzer generates reports about requests processed by a Ruby on Rails server.
+email: narnach@gmail.com
+executables: 
+- rails_analyzer
+extensions: []
+
+extra_rdoc_files: 
+- README.rdoc
+- MIT-LICENSE
+files: 
+- MIT-LICENSE
+- README.rdoc
+- Rakefile
+- rails_analyzer.gemspec
+- spec/rails_analyzer_spec.rb
+- spec/entries_spec.rb
+- spec/url_hits_spec.rb
+- spec/spec.opts
+- spec/spec_helper.rb
+- bin/rails_analyzer
+- lib/rails_analyzer.rb
+- lib/array_ext.rb
+- lib/float_ext.rb
+- lib/time_stats.rb
+- lib/hit_stats.rb
+- lib/entries.rb
+- lib/url_hits.rb
+has_rdoc: true
+homepage: http://www.github.com/Narnach/rails_analyzer
+post_install_message: 
+rdoc_options: 
+- --inline-source
+- --line-numbers
+- --main
+- README.rdoc
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: 1.8.0
+  version: 
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.2.0
+signing_key: 
+specification_version: 2
+summary: RailsAnalyzer generates reports about requests processed by a Ruby on Rails server.
+test_files: 
+- spec/rails_analyzer_spec.rb
+- spec/entries_spec.rb
+- spec/url_hits_spec.rb