request-log-analyzer 1.0.3 → 1.0.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
data/HACKING CHANGED
@@ -1,7 +1,7 @@
1
- HACKING on r-l-a
2
- ----------------
1
+ HACKING on request-log-analyzer
2
+ -------------------------------
3
3
 
4
- - See DESIGN for the basic internal design of r-l-a
4
+ - See DESIGN for the basic internal design of request-log-analyzer
5
5
  - See http://wiki.github.com/wvanbergen/request-log-analyzer/development for
6
6
  more information about developing
7
- - Contact willem AT vanbergen DOT org for any questions
7
+ - Contact me at my GitHub account for any questions: http://github.com/wvanbergen
@@ -0,0 +1,38 @@
1
+ = Request-log-analyzer
2
+
3
+ This is a simple command line tool to analyze request log files of both Rails and
4
+ Merb to produce a performance report. Its purpose is to find what actions are best candidates for optimization.
5
+
6
+ * Analyzes Rails log files (all versions)
7
+ * Can combine multiple files (handy if you are using logrotate)
8
+ * Uses several metrics, including cumulative request time, average request time, process blockers, database and rendering time, HTTP methods and states, Rails action cache statistics, etc.) (Sample output: http://wiki.github.com/wvanbergen/request-log-analyzer/sample-output)
9
+ * Low memory footprint (server-safe)
10
+ * Fast
11
+ * MIT licensed
12
+
13
+ == Installation
14
+
15
+ Install request-log-analyzer as a Ruby gem:
16
+
17
+ $ sudo gem install request-log-analyzer
18
+
19
+ Alternatively, use the gem from the GitHub gem server:
20
+
21
+ $ sudo gem install wvanbergen-request-log-analyzer --source http://gems.github.com
22
+
23
+ To get the best results out of request-log-analyzer, make sure to
24
+ set up logging correctly: http://wiki.github.com/wvanbergen/request-log-analyzer/configure-logging
25
+ for your application.
26
+
27
+ == Usage
28
+
29
+ To analyze a log file and produce a performance report, run request-log-analyzer like this:
30
+
31
+ $ request-log-analyzer log/production.log
32
+
33
+ For more details and available command line options, see the project's wiki:http://wiki.github.com/wvanbergen/request-log-analyzer/basic-usage
34
+
35
+ == Additional information
36
+
37
+ * Project wiki at GitHub: http://wiki.github.com/wvanbergen/request-log-analyzer
38
+ * wvanbergen's blog posts: http://techblog.floorplanner.com/tag/request-log-analyzer
data/Rakefile CHANGED
@@ -1,5 +1,4 @@
1
1
  Dir[File.dirname(__FILE__) + "/tasks/*.rake"].each { |file| load(file) }
2
2
 
3
- desc 'Default: run RSpec for request-log-analyzer.'
4
3
  task :default => :spec
5
4
 
@@ -0,0 +1,114 @@
1
+ class RequestLogAnalyzer::FileFormat::RailsDevelopment < RequestLogAnalyzer::FileFormat
2
+
3
+ # Processing EmployeeController#index (for 123.123.123.123 at 2008-07-13 06:00:00) [GET]
4
+ line_definition :processing do |line|
5
+ line.header = true # this line is the first log line for a request
6
+ line.teaser = /Processing /
7
+ line.regexp = /Processing ((?:\w+::)?\w+)#(\w+)(?: to (\w+))? \(for (\d+\.\d+\.\d+\.\d+) at (\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d)\) \[([A-Z]+)\]/
8
+ line.captures << { :name => :controller, :type => :string } \
9
+ << { :name => :action, :type => :string } \
10
+ << { :name => :format, :type => :string } \
11
+ << { :name => :ip, :type => :string, :anonymize => :ip } \
12
+ << { :name => :timestamp, :type => :timestamp, :anonymize => :slightly } \
13
+ << { :name => :method, :type => :string }
14
+ end
15
+
16
+ # Filter chain halted as [#<ActionController::Caching::Actions::ActionCacheFilter:0x2a999ad620 @check=nil, @options={:store_options=>{}, :layout=>nil, :cache_path=>#<Proc:0x0000002a999b8890@/app/controllers/cached_controller.rb:8>}>] rendered_or_redirected.
17
+ line_definition :cache_hit do |line|
18
+ line.regexp = /Filter chain halted as \[\#<ActionController::Caching::Actions::ActionCacheFilter:.+>\] rendered_or_redirected/
19
+ end
20
+
21
+ # Rendered layouts/_footer (2.9ms)
22
+ line_definition :rendered do |line|
23
+ line.teaser = /Rendered /
24
+ line.regexp = /Rendered (\w+(?:\/\w+)+) \((\d+\.\d+)ms\)/
25
+ line.captures << { :name => :render_file, :type => :string } \
26
+ << { :name => :render_duration, :type => :msec }
27
+ end
28
+
29
+ # User Load (0.4ms) SELECT * FROM `users` WHERE (`users`.`id` = 18205844) 
30
+ line_definition :query_executed do |line|
31
+ line.regexp = /\s+(?:\e\[4;36;1m)?((?:\w+::)*\w+) Load \((\d+\.\d+)ms\)(?:\e\[0m)?\s+(?:\e\[0;1m)?(.+) (?:\e\[0m)?/
32
+ line.captures << { :name => :query_class, :type => :string } \
33
+ << { :name => :query_duration, :type => :msec } \
34
+ << { :name => :query_sql, :type => :string }
35
+ end
36
+
37
+ # CACHE (0.0ms) SELECT * FROM `users` WHERE (`users`.`id` = 0) 
38
+ line_definition :query_cached do |line|
39
+ line.teaser = /\s+(?:\e\[4;35;1m)?CACHE \((\d+\.\d+)ms\)(?:\e\[0m)?\s+(?:\e\[0m)?(.+) (?:\e\[0m)?/
40
+ line.regexp = /\s+(?:\e\[4;35;1m)?CACHE \((\d+\.\d+)ms\)(?:\e\[0m)?\s+(?:\e\[0m)?(.+) (?:\e\[0m)?/
41
+ line.captures << { :name => :cached_duration, :type => :msec } \
42
+ << { :name => :cached_sql, :type => :string }
43
+ end
44
+
45
+ # RuntimeError (Cannot destroy employee): /app/models/employee.rb:198:in `before_destroy'
46
+ line_definition :failed do |line|
47
+ line.footer = true
48
+ line.regexp = /((?:[A-Z]\w+\:\:)*[A-Z]\w+) \((.*)\)(?: on line #(\d+) of .+)?\:(.*)/
49
+ line.captures << { :name => :error, :type => :string } \
50
+ << { :name => :message, :type => :string } \
51
+ << { :name => :line, :type => :integer } \
52
+ << { :name => :file, :type => :string } \
53
+ << { :name => :stack_trace, :type => :string, :anonymize => true }
54
+ end
55
+
56
+
57
+ # Rails < 2.1 completed line example
58
+ # Completed in 0.21665 (4 reqs/sec) | Rendering: 0.00926 (4%) | DB: 0.00000 (0%) | 200 OK [http://demo.nu/employees]
59
+ RAILS_21_COMPLETED = /Completed in (\d+\.\d{5}) \(\d+ reqs\/sec\) (?:\| Rendering: (\d+\.\d{5}) \(\d+\%\) )?(?:\| DB: (\d+\.\d{5}) \(\d+\%\) )?\| (\d\d\d).+\[(http.+)\]/
60
+
61
+ # Rails > 2.1 completed line example
62
+ # Completed in 614ms (View: 120, DB: 31) | 200 OK [http://floorplanner.local/demo]
63
+ RAILS_22_COMPLETED = /Completed in (\d+)ms \((?:View: (\d+), )?DB: (\d+)\) \| (\d\d\d).+\[(http.+)\]/
64
+
65
+ # The completed line uses a kind of hack to ensure that both old style logs and new style logs
66
+ # are both parsed by the same regular expression. The format in Rails 2.2 was slightly changed,
67
+ # but the line contains exactly the same information.
68
+ line_definition :completed do |line|
69
+
70
+ line.footer = true
71
+ line.teaser = /Completed in /
72
+ line.regexp = Regexp.new("(?:#{RAILS_21_COMPLETED}|#{RAILS_22_COMPLETED})")
73
+
74
+ line.captures << { :name => :duration, :type => :sec, :anonymize => :slightly } \
75
+ << { :name => :view, :type => :sec, :anonymize => :slightly } \
76
+ << { :name => :db, :type => :sec, :anonymize => :slightly } \
77
+ << { :name => :status, :type => :integer } \
78
+ << { :name => :url, :type => :string, :anonymize => :url } # Old variant
79
+
80
+ line.captures << { :name => :duration, :type => :msec, :anonymize => :slightly } \
81
+ << { :name => :view, :type => :msec, :anonymize => :slightly } \
82
+ << { :name => :db, :type => :msec, :anonymize => :slightly } \
83
+ << { :name => :status, :type => :integer} \
84
+ << { :name => :url, :type => :string, :anonymize => :url } # 2.2 variant
85
+ end
86
+
87
+
88
+
89
+ REQUEST_CATEGORIZER = Proc.new do |request|
90
+ format = request[:format] || 'html'
91
+ "#{request[:controller]}##{request[:action]}.#{format} [#{request[:method]}]"
92
+ end
93
+
94
+ report do |analyze|
95
+ analyze.timespan :line_type => :processing
96
+ analyze.category :category => REQUEST_CATEGORIZER, :title => 'Top 20 hits', :amount => 20, :line_type => :processing
97
+ analyze.category :method, :title => 'HTTP methods'
98
+ analyze.category :status, :title => 'HTTP statuses returned'
99
+ analyze.category :category => lambda { |request| request =~ :cache_hit ? 'Cache hit' : 'No hit' }, :title => 'Rails action cache hits'
100
+
101
+ analyze.duration :duration, :category => REQUEST_CATEGORIZER, :title => "Request duration", :line_type => :completed
102
+ analyze.duration :view, :category => REQUEST_CATEGORIZER, :title => "Database time", :line_type => :completed
103
+ analyze.duration :db, :category => REQUEST_CATEGORIZER, :title => "View rendering time", :line_type => :completed
104
+
105
+ analyze.category :category => REQUEST_CATEGORIZER, :title => 'Process blockers (> 1 sec duration)',
106
+ :if => lambda { |request| request[:duration] && request[:duration] > 1.0 }, :amount => 20
107
+
108
+ analyze.hourly_spread :line_type => :processing
109
+ analyze.category :error, :title => 'Failed requests', :line_type => :failed, :amount => 20
110
+ end
111
+
112
+
113
+
114
+ end
@@ -73,4 +73,48 @@ describe RequestLogAnalyzer::LogParser, "Rails" do
73
73
  @log_parser.should_receive(:warn).with(:no_current_request, anything).twice
74
74
  @log_parser.parse_file(log_fixture(:rails_unordered))
75
75
  end
76
+ end
77
+
78
+ describe "RequestLogAnalyzer::FileFormat::RailsDevelopment - Rails with development details" do
79
+ include RequestLogAnalyzerSpecHelper
80
+
81
+ before(:each) do
82
+ @file_format = RequestLogAnalyzer::FileFormat.load(:rails_development)
83
+ end
84
+
85
+ it "should have a valid language definitions" do
86
+ @file_format.should be_valid
87
+ end
88
+
89
+ it "should parse a rendered line" do
90
+ info = @file_format.line_definitions[:rendered].matches("Rendered layouts/_footer (2.9ms)")
91
+ info[:render_file].should == 'layouts/_footer'
92
+ info[:render_duration].should == 0.0029
93
+ end
94
+
95
+ it "should parse a query executed line with colors" do
96
+ info = @file_format.line_definitions[:query_executed].matches(" User Load (0.4ms) SELECT * FROM `users` WHERE (`users`.`id` = 18205844) ")
97
+ info[:query_class].should == 'User'
98
+ info[:query_duration].should == 0.0004
99
+ info[:query_sql].should == 'SELECT * FROM `users` WHERE (`users`.`id` = 18205844)'
100
+ end
101
+
102
+ it "should parse a query executed line without colors" do
103
+ info = @file_format.line_definitions[:query_executed].matches(" User Load (0.4ms) SELECT * FROM `users` WHERE (`users`.`id` = 18205844) ")
104
+ info[:query_class].should == 'User'
105
+ info[:query_duration].should == 0.0004
106
+ info[:query_sql].should == 'SELECT * FROM `users` WHERE (`users`.`id` = 18205844)'
107
+ end
108
+
109
+ it "should parse a cached query line with colors" do
110
+ info = @file_format.line_definitions[:query_cached].matches(' CACHE (0.0ms) SELECT * FROM `users` WHERE (`users`.`id` = 0) ')
111
+ info[:cached_duration].should == 0.0
112
+ info[:cached_sql].should == 'SELECT * FROM `users` WHERE (`users`.`id` = 0)'
113
+ end
114
+
115
+ it "should parse a cached query line without colors" do
116
+ info = @file_format.line_definitions[:query_cached].matches(' CACHE (0.0ms) SELECT * FROM `users` WHERE (`users`.`id` = 0) ')
117
+ info[:cached_duration].should == 0.0
118
+ info[:cached_sql].should == 'SELECT * FROM `users` WHERE (`users`.`id` = 0)'
119
+ end
76
120
  end
@@ -2,7 +2,6 @@ require 'rubygems'
2
2
  require 'rubyforge'
3
3
  require 'rake'
4
4
  require 'rake/tasklib'
5
- require 'rake/gempackagetask'
6
5
  require 'date'
7
6
 
8
7
  module Rake
@@ -26,13 +25,56 @@ module Rake
26
25
  namespace(:gem) do
27
26
  desc "Updates the file lists for this gem"
28
27
  task(:manifest) { manifest_task }
28
+
29
+ desc "Releases a new version of #{@name}"
30
+ task(:build => [:manifest]) { build_task }
29
31
 
30
32
  desc "Releases a new version of #{@name}"
31
- task(:release => :package) { release_task }
33
+ task(:release => [:check_clean_master_branch, :version, :build]) { release_task }
34
+
35
+ # helper task for releasing
36
+ task(:check_clean_master_branch) { verify_clean_status('master') }
37
+ task(:check_version) { verify_version(ENV['VERSION'] || @specification.version) }
38
+ task(:version => [:check_version]) { set_gem_version! }
39
+ end
40
+
41
+ # Register RDoc tasks
42
+ if @specification.has_rdoc
43
+ require 'rake/rdoctask'
32
44
 
33
- Rake::GemPackageTask.new(@specification) do |pkg|
45
+ namespace(:doc) do
46
+ desc 'Generate documentation for request-log-analyzer'
47
+ Rake::RDocTask.new(:compile) do |rdoc|
48
+ rdoc.rdoc_dir = 'doc'
49
+ rdoc.title = @name
50
+ rdoc.options += @specification.rdoc_options
51
+ rdoc.rdoc_files.include(@specification.extra_rdoc_files)
52
+ rdoc.rdoc_files.include('lib/**/*.rb')
53
+ end
54
+ end
55
+ end
56
+
57
+ # Setup :spec task if RSpec files exist
58
+ if Dir['spec/**/*_spec.rb'].any?
59
+ require 'spec/rake/spectask'
60
+
61
+ desc "Run all specs for #{@name}"
62
+ Spec::Rake::SpecTask.new(:spec) do |t|
63
+ t.spec_files = FileList['spec/**/*_spec.rb']
64
+ end
65
+ end
66
+
67
+ # Setup :test task if unit test files exist
68
+ if Dir['test/**/*_test.rb'].any?
69
+ require 'rake/testtask'
70
+
71
+ desc "Run all unit tests for #{@name}"
72
+ Rake::TestTask.new(:test) do |t|
73
+ t.pattern = 'test/**/*_test.rb'
74
+ t.verbose = true
75
+ t.libs << 'test'
34
76
  end
35
- end
77
+ end
36
78
  end
37
79
 
38
80
  protected
@@ -110,6 +152,12 @@ module Rake
110
152
  newest_version = run_command('git tag').map { |tag| tag.split(name + '-').last }.compact.map { |v| Gem::Version.new(v) }.max
111
153
  raise "This version number (#{new_version}) is not higher than the highest tagged version (#{newest_version})" if !newest_version.nil? && newest_version >= Gem::Version.new(new_version.to_s)
112
154
  end
155
+
156
+ def set_gem_version!
157
+ # update gemspec file
158
+ self.gemspec_version = ENV['VERSION'] if Gem::Version.correct?(ENV['VERSION'])
159
+ self.gemspec_date = Date.today
160
+ end
113
161
 
114
162
  def manifest_task
115
163
  verify_current_branch('master')
@@ -139,26 +187,22 @@ module Rake
139
187
 
140
188
  def build_task
141
189
  sh "gem build #{gemspec_file}"
190
+ Dir.mkdir('pkg') unless File.exist?('pkg')
191
+ sh "mv #{name}-#{specification.version}.gem pkg/#{name}-#{specification.version}.gem"
142
192
  end
143
193
 
144
194
  def install_task
145
- raise "#{name} .gem file not found" unless File.exist?("#{name}-#{specification.version}.gem")
146
- sh "gem install #{name}-#{specification.version}.gem"
195
+ raise "#{name} .gem file not found" unless File.exist?("pkg/#{name}-#{specification.version}.gem")
196
+ sh "gem install pkg/#{name}-#{specification.version}.gem"
147
197
  end
148
198
 
149
199
  def uninstall_task
150
- raise "#{name} .gem file not found" unless File.exist?("#{name}-#{specification.version}.gem")
200
+ raise "#{name} .gem file not found" unless File.exist?("pkg/#{name}-#{specification.version}.gem")
151
201
  sh "gem uninstall #{name}"
152
202
  end
153
203
 
154
204
  def release_task
155
- verify_clean_status('master')
156
- verify_version(ENV['VERSION'] || @specification.version)
157
-
158
- # update gemspec file
159
- self.gemspec_version = ENV['VERSION'] if Gem::Version.correct?(ENV['VERSION'])
160
- self.gemspec_date = Date.today
161
- manifest_task
205
+ # commit the gemspec file
162
206
  git_commit_file(gemspec_file, "Updated #{gemspec_file} for release of version #{@specification.version}") if git_modified?(gemspec_file)
163
207
 
164
208
  # create tag and push changes
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: request-log-analyzer
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.0.3
4
+ version: 1.0.4
5
5
  platform: ruby
6
6
  authors:
7
7
  - Willem van Bergen
@@ -10,7 +10,7 @@ autorequire:
10
10
  bindir: bin
11
11
  cert_chain: []
12
12
 
13
- date: 2009-01-14 00:00:00 +01:00
13
+ date: 2009-01-17 00:00:00 +01:00
14
14
  default_executable: request-log-analyzer
15
15
  dependencies: []
16
16
 
@@ -20,13 +20,13 @@ executables:
20
20
  - request-log-analyzer
21
21
  extensions: []
22
22
 
23
- extra_rdoc_files: []
24
-
23
+ extra_rdoc_files:
24
+ - README.rdoc
25
25
  files:
26
26
  - DESIGN
27
27
  - HACKING
28
28
  - LICENSE
29
- - README.textile
29
+ - README.rdoc
30
30
  - Rakefile
31
31
  - bin
32
32
  - bin/request-log-analyzer
@@ -47,6 +47,7 @@ files:
47
47
  - lib/request_log_analyzer/file_format.rb
48
48
  - lib/request_log_analyzer/file_format/merb.rb
49
49
  - lib/request_log_analyzer/file_format/rails.rb
50
+ - lib/request_log_analyzer/file_format/rails_development.rb
50
51
  - lib/request_log_analyzer/filter
51
52
  - lib/request_log_analyzer/filter/anonimize.rb
52
53
  - lib/request_log_analyzer/filter/base.rb
@@ -95,12 +96,16 @@ files:
95
96
  - tasks
96
97
  - tasks/github-gem.rake
97
98
  - tasks/request_log_analyzer.rake
98
- - tasks/rspec.rake
99
- has_rdoc: false
99
+ has_rdoc: true
100
100
  homepage: http://github.com/wvanbergen/request-log-analyzer/wikis
101
101
  post_install_message:
102
- rdoc_options: []
103
-
102
+ rdoc_options:
103
+ - --title
104
+ - request-log-analyzer
105
+ - --main
106
+ - README.rdoc
107
+ - --line-numbers
108
+ - --inline-source
104
109
  require_paths:
105
110
  - lib
106
111
  required_ruby_version: !ruby/object:Gem::Requirement
@@ -118,7 +123,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
118
123
  requirements: []
119
124
 
120
125
  rubyforge_project: r-l-a
121
- rubygems_version: 1.3.1
126
+ rubygems_version: 1.2.0
122
127
  signing_key:
123
128
  specification_version: 2
124
129
  summary: A command line tool to analyze Rails logs
@@ -1,36 +0,0 @@
1
- h1. Request-log-analyzer
2
-
3
- This is a simple command line tool to analyze request log files of both Rails and
4
- Merb to produce a performance report. Its purpose is to find what actions are best candidates for optimization.
5
-
6
- * Analyzes Rails log files (all versions)
7
- * Can combine multiple files (handy if you are using logrotate)
8
- * Uses several metrics, including cumulative request time, average request time, process blockers, database and rendering time, HTTP methods and states, Rails action cache statistics, etc.) ("Sample output":http://wiki.github.com/wvanbergen/request-log-analyzer/sample-output)
9
- * Low memory footprint (server-safe)
10
- * Fast
11
- * MIT licensed
12
-
13
- h2. Installation
14
-
15
- <pre>
16
- $ sudo gem install wvanbergen-request-log-analyzer --source http://gems.github.com
17
- </pre>
18
-
19
- To get the best results out of request-log-analyzer, make sure to
20
- "set up logging correctly":http://wiki.github.com/wvanbergen/request-log-analyzer/configure-logging
21
- for your application.
22
-
23
- h2. Usage
24
-
25
- To analyze a log file and produce a performance report, run request-log-analyzer like this:
26
-
27
- <pre>
28
- $ request-log-analyzer log/production.log
29
- </pre>
30
-
31
- For more details and available command line options, see the "project's wiki":http://wiki.github.com/wvanbergen/request-log-analyzer/basic-usage
32
-
33
- h2. Additional information
34
-
35
- * "Project wiki at GitHub":http://wiki.github.com/wvanbergen/request-log-analyzer
36
- * "wvanbergen's blog posts":http://techblog.floorplanner.com/tag/request-log-analyzer/
@@ -1,6 +0,0 @@
1
- require 'spec/rake/spectask'
2
-
3
- desc "Run all specs in spec directory (excluding plugin specs)"
4
- Spec::Rake::SpecTask.new(:spec) do |t|
5
- t.spec_files = FileList['spec/**/*_spec.rb']
6
- end