wvanbergen-request-log-analyzer 1.0.3 → 1.0.4

data/DESIGN

@@ -12,3 +12,13 @@ This allows you to easily add extra reports, filters and outputs.
 
 3) Gather output from pipeline.
   Controller.report
+
+At the moment the supported sources are file and STDIN.
+In the future we want to be able to have a generated request database as source.
+This will make interactive downdrilling possible.
+
+
+For the report generation output we now use the File and the STDOUT class, as they both support <<.
+In the future we want to have a OutputFile, OutputSTDOUT and OutputHTML class, so that reports can generate
+tables, lines and comments and push them into the output class.
+

data/HACKING

@@ -1,7 +1,7 @@
-HACKING on r-l-a
-----------------
+HACKING on request-log-analyzer
+-------------------------------
 
-- See DESIGN for the basic internal design of r-l-a
+- See DESIGN for the basic internal design of request-log-analyzer
 - See http://wiki.github.com/wvanbergen/request-log-analyzer/development for
   more information about developing
-- Contact willem AT vanbergen DOT org for any questions
+- Contact me at my GitHub account for any questions: http://github.com/wvanbergen

data/README.rdoc

@@ -0,0 +1,38 @@
+= Request-log-analyzer
+
+This is a simple command line tool to analyze request log files of both Rails and
+Merb to produce a performance report. Its purpose is to find what actions are best candidates for optimization.
+
+* Analyzes Rails log files (all versions)
+* Can combine multiple files (handy if you are using logrotate)
+* Uses several metrics, including cumulative request time, average request time, process blockers, database and rendering time, HTTP methods and states, Rails action cache statistics, etc.) (Sample output: http://wiki.github.com/wvanbergen/request-log-analyzer/sample-output)
+* Low memory footprint (server-safe)
+* Fast
+* MIT licensed
+
+== Installation
+
+Install request-log-analyzer as a Ruby gem:
+
+  $ sudo gem install request-log-analyzer
+
+Alternatively, use the gem from the GitHub gem server:
+
+  $ sudo gem install wvanbergen-request-log-analyzer --source http://gems.github.com
+
+To get the best results out of request-log-analyzer, make sure to 
+set up logging correctly: http://wiki.github.com/wvanbergen/request-log-analyzer/configure-logging
+for your application.
+
+== Usage
+
+To analyze a log file and produce a performance report, run request-log-analyzer like this:
+
+  $ request-log-analyzer log/production.log
+
+For more details and available command line options, see the project's wiki:http://wiki.github.com/wvanbergen/request-log-analyzer/basic-usage
+
+== Additional information
+
+* Project wiki at GitHub: http://wiki.github.com/wvanbergen/request-log-analyzer
+* wvanbergen's blog posts: http://techblog.floorplanner.com/tag/request-log-analyzer

data/Rakefile

@@ -1,5 +1,4 @@
 Dir[File.dirname(__FILE__) + "/tasks/*.rake"].each { |file| load(file) }
  
-desc 'Default: run RSpec for request-log-analyzer.'
 task :default => :spec
 

data/lib/request_log_analyzer/file_format/rails_development.rb

@@ -0,0 +1,114 @@
+class RequestLogAnalyzer::FileFormat::RailsDevelopment < RequestLogAnalyzer::FileFormat
+  
+  # Processing EmployeeController#index (for 123.123.123.123 at 2008-07-13 06:00:00) [GET]
+  line_definition :processing do |line|
+    line.header = true # this line is the first log line for a request 
+    line.teaser = /Processing /
+    line.regexp = /Processing ((?:\w+::)?\w+)#(\w+)(?: to (\w+))? \(for (\d+\.\d+\.\d+\.\d+) at (\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d)\) \[([A-Z]+)\]/
+    line.captures << { :name => :controller, :type  => :string } \
+                  << { :name => :action,     :type  => :string } \
+                  << { :name => :format,     :type  => :string } \
+                  << { :name => :ip,         :type  => :string, :anonymize => :ip } \
+                  << { :name => :timestamp,  :type  => :timestamp, :anonymize => :slightly } \
+                  << { :name => :method,     :type  => :string }
+  end
+
+  # Filter chain halted as [#<ActionController::Caching::Actions::ActionCacheFilter:0x2a999ad620 @check=nil, @options={:store_options=>{}, :layout=>nil, :cache_path=>#<Proc:0x0000002a999b8890@/app/controllers/cached_controller.rb:8>}>] rendered_or_redirected.    
+  line_definition :cache_hit do |line|
+    line.regexp = /Filter chain halted as \[\#<ActionController::Caching::Actions::ActionCacheFilter:.+>\] rendered_or_redirected/
+  end
+  
+  # Rendered layouts/_footer (2.9ms)
+  line_definition :rendered do |line|
+    line.teaser = /Rendered /
+    line.regexp = /Rendered (\w+(?:\/\w+)+) \((\d+\.\d+)ms\)/
+    line.captures << { :name => :render_file,     :type  => :string } \
+                  << { :name => :render_duration, :type  => :msec }
+  end
+  
+  #  User Load (0.4ms)   SELECT * FROM `users` WHERE (`users`.`id` = 18205844) 
+  line_definition :query_executed do |line|
+    line.regexp = /\s+(?:\e\[4;36;1m)?((?:\w+::)*\w+) Load \((\d+\.\d+)ms\)(?:\e\[0m)?\s+(?:\e\[0;1m)?(.+) (?:\e\[0m)?/
+    line.captures << { :name => :query_class,    :type  => :string } \
+                  << { :name => :query_duration, :type  => :msec } \
+                  << { :name => :query_sql,      :type  => :string }
+  end
+
+  #  CACHE (0.0ms)   SELECT * FROM `users` WHERE (`users`.`id` = 0)   
+  line_definition :query_cached do |line|
+    line.teaser = /\s+(?:\e\[4;35;1m)?CACHE \((\d+\.\d+)ms\)(?:\e\[0m)?\s+(?:\e\[0m)?(.+) (?:\e\[0m)?/  
+    line.regexp = /\s+(?:\e\[4;35;1m)?CACHE \((\d+\.\d+)ms\)(?:\e\[0m)?\s+(?:\e\[0m)?(.+) (?:\e\[0m)?/
+    line.captures << { :name => :cached_duration, :type  => :msec } \
+                  << { :name => :cached_sql,      :type  => :string }
+  end  
+  
+  # RuntimeError (Cannot destroy employee):  /app/models/employee.rb:198:in `before_destroy' 
+  line_definition :failed do |line|
+    line.footer = true
+    line.regexp = /((?:[A-Z]\w+\:\:)*[A-Z]\w+) \((.*)\)(?: on line #(\d+) of .+)?\:(.*)/
+    line.captures << { :name => :error,       :type => :string } \
+                  << { :name => :message,     :type => :string } \
+                  << { :name => :line,        :type => :integer } \
+                  << { :name => :file,        :type => :string } \
+                  << { :name => :stack_trace, :type => :string, :anonymize => true }
+  end
+
+
+  # Rails < 2.1 completed line example
+  # Completed in 0.21665 (4 reqs/sec) | Rendering: 0.00926 (4%) | DB: 0.00000 (0%) | 200 OK [http://demo.nu/employees]
+  RAILS_21_COMPLETED = /Completed in (\d+\.\d{5}) \(\d+ reqs\/sec\) (?:\| Rendering: (\d+\.\d{5}) \(\d+\%\) )?(?:\| DB: (\d+\.\d{5}) \(\d+\%\) )?\| (\d\d\d).+\[(http.+)\]/
+
+  # Rails > 2.1 completed line example
+  # Completed in 614ms (View: 120, DB: 31) | 200 OK [http://floorplanner.local/demo]  
+  RAILS_22_COMPLETED = /Completed in (\d+)ms \((?:View: (\d+), )?DB: (\d+)\) \| (\d\d\d).+\[(http.+)\]/
+
+  # The completed line uses a kind of hack to ensure that both old style logs and new style logs 
+  # are both parsed by the same regular expression. The format in Rails 2.2 was slightly changed,
+  # but the line contains exactly the same information.
+  line_definition :completed do |line|
+    
+    line.footer = true
+    line.teaser = /Completed in /
+    line.regexp = Regexp.new("(?:#{RAILS_21_COMPLETED}|#{RAILS_22_COMPLETED})")
+    
+    line.captures << { :name => :duration, :type => :sec,    :anonymize => :slightly } \
+                  << { :name => :view,     :type => :sec,    :anonymize => :slightly } \
+                  << { :name => :db,       :type => :sec,    :anonymize => :slightly } \
+                  << { :name => :status,   :type => :integer } \
+                  << { :name => :url,      :type => :string, :anonymize => :url } # Old variant 
+                  
+    line.captures << { :name => :duration, :type => :msec,   :anonymize => :slightly } \
+                  << { :name => :view,     :type => :msec,   :anonymize => :slightly } \
+                  << { :name => :db,       :type => :msec,   :anonymize => :slightly } \
+                  << { :name => :status,   :type => :integer} \
+                  << { :name => :url,      :type => :string, :anonymize => :url }  # 2.2 variant 
+  end
+
+
+
+  REQUEST_CATEGORIZER = Proc.new do |request|
+    format = request[:format] || 'html'
+    "#{request[:controller]}##{request[:action]}.#{format} [#{request[:method]}]"
+  end
+
+  report do |analyze|
+    analyze.timespan :line_type => :processing
+    analyze.category :category => REQUEST_CATEGORIZER, :title => 'Top 20 hits', :amount => 20, :line_type => :processing
+    analyze.category :method, :title => 'HTTP methods'
+    analyze.category :status, :title => 'HTTP statuses returned'
+    analyze.category :category => lambda { |request| request =~ :cache_hit ? 'Cache hit' : 'No hit' }, :title => 'Rails action cache hits'
+    
+    analyze.duration :duration, :category => REQUEST_CATEGORIZER, :title => "Request duration",    :line_type => :completed
+    analyze.duration :view,     :category => REQUEST_CATEGORIZER, :title => "Database time",       :line_type => :completed
+    analyze.duration :db,       :category => REQUEST_CATEGORIZER, :title => "View rendering time", :line_type => :completed
+    
+    analyze.category :category => REQUEST_CATEGORIZER, :title => 'Process blockers (> 1 sec duration)', 
+            :if => lambda { |request| request[:duration] && request[:duration] > 1.0 }, :amount => 20
+            
+    analyze.hourly_spread :line_type => :processing
+    analyze.category :error, :title => 'Failed requests', :line_type => :failed, :amount => 20
+  end
+
+
+
+end

data/spec/rails_format_spec.rb

@@ -73,4 +73,48 @@ describe RequestLogAnalyzer::LogParser, "Rails" do
     @log_parser.should_receive(:warn).with(:no_current_request, anything).twice
     @log_parser.parse_file(log_fixture(:rails_unordered))
   end  
+end
+
+describe "RequestLogAnalyzer::FileFormat::RailsDevelopment - Rails with development details" do
+  include RequestLogAnalyzerSpecHelper
+  
+  before(:each) do
+    @file_format = RequestLogAnalyzer::FileFormat.load(:rails_development)
+  end
+  
+  it "should have a valid language definitions" do
+    @file_format.should be_valid
+  end
+  
+  it "should parse a rendered line" do
+    info = @file_format.line_definitions[:rendered].matches("Rendered layouts/_footer (2.9ms)")
+    info[:render_file].should == 'layouts/_footer'
+    info[:render_duration].should == 0.0029
+  end
+  
+  it "should parse a query executed line with colors" do
+    info = @file_format.line_definitions[:query_executed].matches(" User Load (0.4ms)   SELECT * FROM `users` WHERE (`users`.`id` = 18205844) ")
+    info[:query_class].should == 'User'
+    info[:query_duration].should == 0.0004
+    info[:query_sql].should == 'SELECT * FROM `users` WHERE (`users`.`id` = 18205844)'
+  end  
+  
+  it "should parse a query executed line without colors" do
+    info = @file_format.line_definitions[:query_executed].matches(" User Load (0.4ms)   SELECT * FROM `users` WHERE (`users`.`id` = 18205844) ")
+    info[:query_class].should == 'User'
+    info[:query_duration].should == 0.0004
+    info[:query_sql].should == 'SELECT * FROM `users` WHERE (`users`.`id` = 18205844)'    
+  end  
+  
+  it "should parse a cached query line with colors" do
+    info = @file_format.line_definitions[:query_cached].matches(' CACHE (0.0ms)   SELECT * FROM `users` WHERE (`users`.`id` = 0) ')
+    info[:cached_duration].should == 0.0
+    info[:cached_sql].should == 'SELECT * FROM `users` WHERE (`users`.`id` = 0)'   
+  end
+  
+  it "should parse a cached query line without colors" do
+    info = @file_format.line_definitions[:query_cached].matches(' CACHE (0.0ms)   SELECT * FROM `users` WHERE (`users`.`id` = 0) ')
+    info[:cached_duration].should == 0.0
+    info[:cached_sql].should == 'SELECT * FROM `users` WHERE (`users`.`id` = 0)'   
+  end  
 end

data/tasks/github-gem.rake

@@ -2,7 +2,6 @@ require 'rubygems'
 require 'rubyforge'
 require 'rake'
 require 'rake/tasklib'
-require 'rake/gempackagetask'
 require 'date'
 
 module Rake 
@@ -26,13 +25,56 @@ module Rake
       namespace(:gem) do
         desc "Updates the file lists for this gem"
         task(:manifest) { manifest_task }
+
+        desc "Releases a new version of #{@name}"
+        task(:build => [:manifest]) { build_task } 
         
         desc "Releases a new version of #{@name}"
-        task(:release => :package) { release_task } 
+        task(:release => [:check_clean_master_branch, :version, :build]) { release_task } 
+        
+        # helper task for releasing
+        task(:check_clean_master_branch) { verify_clean_status('master') }
+        task(:check_version) { verify_version(ENV['VERSION'] || @specification.version) }
+        task(:version => [:check_version]) { set_gem_version! }        
+      end
+      
+      # Register RDoc tasks
+      if @specification.has_rdoc
+        require 'rake/rdoctask'
         
-        Rake::GemPackageTask.new(@specification) do |pkg|
+        namespace(:doc) do 
+          desc 'Generate documentation for request-log-analyzer'
+          Rake::RDocTask.new(:compile) do |rdoc|
+            rdoc.rdoc_dir = 'doc'
+            rdoc.title    = @name
+            rdoc.options += @specification.rdoc_options
+            rdoc.rdoc_files.include(@specification.extra_rdoc_files)
+            rdoc.rdoc_files.include('lib/**/*.rb')
+          end
+        end
+      end
+    
+      # Setup :spec task if RSpec files exist
+      if Dir['spec/**/*_spec.rb'].any?
+        require 'spec/rake/spectask'
+
+        desc "Run all specs for #{@name}"
+        Spec::Rake::SpecTask.new(:spec) do |t|
+          t.spec_files = FileList['spec/**/*_spec.rb']
+        end
+      end
+      
+      # Setup :test task if unit test files exist
+      if Dir['test/**/*_test.rb'].any?
+        require 'rake/testtask'
+
+        desc "Run all unit tests for #{@name}"
+        Rake::TestTask.new(:test) do |t|
+          t.pattern = 'test/**/*_test.rb'
+          t.verbose = true
+          t.libs << 'test'
         end
-      end    
+      end      
     end
     
     protected 
@@ -110,6 +152,12 @@ module Rake
       newest_version = run_command('git tag').map { |tag| tag.split(name + '-').last }.compact.map { |v| Gem::Version.new(v) }.max
       raise "This version number (#{new_version}) is not higher than the highest tagged version (#{newest_version})" if !newest_version.nil? && newest_version >= Gem::Version.new(new_version.to_s)
     end
+    
+    def set_gem_version!
+      # update gemspec file
+      self.gemspec_version = ENV['VERSION'] if Gem::Version.correct?(ENV['VERSION'])
+      self.gemspec_date    = Date.today
+    end
 
     def manifest_task
       verify_current_branch('master')
@@ -139,26 +187,22 @@ module Rake
     
     def build_task
       sh "gem build #{gemspec_file}"
+      Dir.mkdir('pkg') unless File.exist?('pkg')
+      sh "mv #{name}-#{specification.version}.gem pkg/#{name}-#{specification.version}.gem" 
     end
     
     def install_task
-      raise "#{name} .gem file not found" unless File.exist?("#{name}-#{specification.version}.gem")
-      sh "gem install #{name}-#{specification.version}.gem"
+      raise "#{name} .gem file not found" unless File.exist?("pkg/#{name}-#{specification.version}.gem")
+      sh "gem install pkg/#{name}-#{specification.version}.gem"
     end
     
     def uninstall_task
-      raise "#{name} .gem file not found" unless File.exist?("#{name}-#{specification.version}.gem")
+      raise "#{name} .gem file not found" unless File.exist?("pkg/#{name}-#{specification.version}.gem")
       sh "gem uninstall #{name}"
     end    
     
     def release_task
-      verify_clean_status('master')
-      verify_version(ENV['VERSION'] || @specification.version)
-      
-      # update gemspec file
-      self.gemspec_version = ENV['VERSION'] if Gem::Version.correct?(ENV['VERSION'])
-      self.gemspec_date    = Date.today
-      manifest_task      
+      # commit the gemspec file
       git_commit_file(gemspec_file, "Updated #{gemspec_file} for release of version #{@specification.version}") if git_modified?(gemspec_file)
 
       # create tag and push changes

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification 
 name: wvanbergen-request-log-analyzer
 version: !ruby/object:Gem::Version 
-  version: 1.0.3
+  version: 1.0.4
 platform: ruby
 authors: 
 - Willem van Bergen
@@ -10,7 +10,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2009-01-14 00:00:00 -08:00
+date: 2009-01-17 00:00:00 -08:00
 default_executable: request-log-analyzer
 dependencies: []
 
@@ -20,13 +20,13 @@ executables:
 - request-log-analyzer
 extensions: []
 
-extra_rdoc_files: []
-
+extra_rdoc_files: 
+- README.rdoc
 files: 
 - DESIGN
 - HACKING
 - LICENSE
-- README.textile
+- README.rdoc
 - Rakefile
 - bin
 - bin/request-log-analyzer
@@ -47,6 +47,7 @@ files:
 - lib/request_log_analyzer/file_format.rb
 - lib/request_log_analyzer/file_format/merb.rb
 - lib/request_log_analyzer/file_format/rails.rb
+- lib/request_log_analyzer/file_format/rails_development.rb
 - lib/request_log_analyzer/filter
 - lib/request_log_analyzer/filter/anonimize.rb
 - lib/request_log_analyzer/filter/base.rb
@@ -95,12 +96,16 @@ files:
 - tasks
 - tasks/github-gem.rake
 - tasks/request_log_analyzer.rake
-- tasks/rspec.rake
-has_rdoc: false
+has_rdoc: true
 homepage: http://github.com/wvanbergen/request-log-analyzer/wikis
 post_install_message: 
-rdoc_options: []
-
+rdoc_options: 
+- --title
+- request-log-analyzer
+- --main
+- README.rdoc
+- --line-numbers
+- --inline-source
 require_paths: 
 - lib
 required_ruby_version: !ruby/object:Gem::Requirement 

data/README.textile

@@ -1,36 +0,0 @@
-h1. Request-log-analyzer
-
-This is a simple command line tool to analyze request log files of both Rails and
-Merb to produce a performance report. Its purpose is to find what actions are best candidates for optimization.
-
-* Analyzes Rails log files (all versions)
-* Can combine multiple files (handy if you are using logrotate)
-* Uses several metrics, including cumulative request time, average request time, process blockers, database and rendering time, HTTP methods and states, Rails action cache statistics, etc.) ("Sample output":http://wiki.github.com/wvanbergen/request-log-analyzer/sample-output)
-* Low memory footprint (server-safe)
-* Fast
-* MIT licensed
-
-h2. Installation
-
-<pre>
-$ sudo gem install wvanbergen-request-log-analyzer --source http://gems.github.com
-</pre>
-
-To get the best results out of request-log-analyzer, make sure to 
-"set up logging correctly":http://wiki.github.com/wvanbergen/request-log-analyzer/configure-logging
-for your application.
-
-h2. Usage
-
-To analyze a log file and produce a performance report, run request-log-analyzer like this:
-
-<pre>
-$ request-log-analyzer log/production.log
-</pre>
-
-For more details and available command line options, see the "project's wiki":http://wiki.github.com/wvanbergen/request-log-analyzer/basic-usage
-
-h2. Additional information
-
-* "Project wiki at GitHub":http://wiki.github.com/wvanbergen/request-log-analyzer
-* "wvanbergen's blog posts":http://techblog.floorplanner.com/tag/request-log-analyzer/

data/tasks/rspec.rake

@@ -1,6 +0,0 @@
-require 'spec/rake/spectask'
- 
-desc "Run all specs in spec directory (excluding plugin specs)"
-Spec::Rake::SpecTask.new(:spec) do |t|
-  t.spec_files = FileList['spec/**/*_spec.rb']
-end