wvanbergen-request-log-analyzer 1.0.3 → 1.0.4

Sign up to get free protection for your applications and to get access to all the features.
data/DESIGN CHANGED
@@ -12,3 +12,13 @@ This allows you to easily add extra reports, filters and outputs.
12
12
 
13
13
  3) Gather output from pipeline.
14
14
  Controller.report
15
+
16
+ At the moment the supported sources are file and STDIN.
17
+ In the future we want to be able to have a generated request database as source.
18
+ This will make interactive downdrilling possible.
19
+
20
+
21
+ For the report generation output we now use the File and the STDOUT class, as they both support <<.
22
+ In the future we want to have a OutputFile, OutputSTDOUT and OutputHTML class, so that reports can generate
23
+ tables, lines and comments and push them into the output class.
24
+
data/HACKING CHANGED
@@ -1,7 +1,7 @@
1
- HACKING on r-l-a
2
- ----------------
1
+ HACKING on request-log-analyzer
2
+ -------------------------------
3
3
 
4
- - See DESIGN for the basic internal design of r-l-a
4
+ - See DESIGN for the basic internal design of request-log-analyzer
5
5
  - See http://wiki.github.com/wvanbergen/request-log-analyzer/development for
6
6
  more information about developing
7
- - Contact willem AT vanbergen DOT org for any questions
7
+ - Contact me at my GitHub account for any questions: http://github.com/wvanbergen
data/README.rdoc ADDED
@@ -0,0 +1,38 @@
1
+ = Request-log-analyzer
2
+
3
+ This is a simple command line tool to analyze request log files of both Rails and
4
+ Merb to produce a performance report. Its purpose is to find what actions are best candidates for optimization.
5
+
6
+ * Analyzes Rails log files (all versions)
7
+ * Can combine multiple files (handy if you are using logrotate)
8
+ * Uses several metrics, including cumulative request time, average request time, process blockers, database and rendering time, HTTP methods and states, Rails action cache statistics, etc.) (Sample output: http://wiki.github.com/wvanbergen/request-log-analyzer/sample-output)
9
+ * Low memory footprint (server-safe)
10
+ * Fast
11
+ * MIT licensed
12
+
13
+ == Installation
14
+
15
+ Install request-log-analyzer as a Ruby gem:
16
+
17
+ $ sudo gem install request-log-analyzer
18
+
19
+ Alternatively, use the gem from the GitHub gem server:
20
+
21
+ $ sudo gem install wvanbergen-request-log-analyzer --source http://gems.github.com
22
+
23
+ To get the best results out of request-log-analyzer, make sure to
24
+ set up logging correctly: http://wiki.github.com/wvanbergen/request-log-analyzer/configure-logging
25
+ for your application.
26
+
27
+ == Usage
28
+
29
+ To analyze a log file and produce a performance report, run request-log-analyzer like this:
30
+
31
+ $ request-log-analyzer log/production.log
32
+
33
+ For more details and available command line options, see the project's wiki:http://wiki.github.com/wvanbergen/request-log-analyzer/basic-usage
34
+
35
+ == Additional information
36
+
37
+ * Project wiki at GitHub: http://wiki.github.com/wvanbergen/request-log-analyzer
38
+ * wvanbergen's blog posts: http://techblog.floorplanner.com/tag/request-log-analyzer
data/Rakefile CHANGED
@@ -1,5 +1,4 @@
1
1
  Dir[File.dirname(__FILE__) + "/tasks/*.rake"].each { |file| load(file) }
2
2
 
3
- desc 'Default: run RSpec for request-log-analyzer.'
4
3
  task :default => :spec
5
4
 
@@ -0,0 +1,114 @@
1
+ class RequestLogAnalyzer::FileFormat::RailsDevelopment < RequestLogAnalyzer::FileFormat
2
+
3
+ # Processing EmployeeController#index (for 123.123.123.123 at 2008-07-13 06:00:00) [GET]
4
+ line_definition :processing do |line|
5
+ line.header = true # this line is the first log line for a request
6
+ line.teaser = /Processing /
7
+ line.regexp = /Processing ((?:\w+::)?\w+)#(\w+)(?: to (\w+))? \(for (\d+\.\d+\.\d+\.\d+) at (\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d)\) \[([A-Z]+)\]/
8
+ line.captures << { :name => :controller, :type => :string } \
9
+ << { :name => :action, :type => :string } \
10
+ << { :name => :format, :type => :string } \
11
+ << { :name => :ip, :type => :string, :anonymize => :ip } \
12
+ << { :name => :timestamp, :type => :timestamp, :anonymize => :slightly } \
13
+ << { :name => :method, :type => :string }
14
+ end
15
+
16
+ # Filter chain halted as [#<ActionController::Caching::Actions::ActionCacheFilter:0x2a999ad620 @check=nil, @options={:store_options=>{}, :layout=>nil, :cache_path=>#<Proc:0x0000002a999b8890@/app/controllers/cached_controller.rb:8>}>] rendered_or_redirected.
17
+ line_definition :cache_hit do |line|
18
+ line.regexp = /Filter chain halted as \[\#<ActionController::Caching::Actions::ActionCacheFilter:.+>\] rendered_or_redirected/
19
+ end
20
+
21
+ # Rendered layouts/_footer (2.9ms)
22
+ line_definition :rendered do |line|
23
+ line.teaser = /Rendered /
24
+ line.regexp = /Rendered (\w+(?:\/\w+)+) \((\d+\.\d+)ms\)/
25
+ line.captures << { :name => :render_file, :type => :string } \
26
+ << { :name => :render_duration, :type => :msec }
27
+ end
28
+
29
+ # User Load (0.4ms) SELECT * FROM `users` WHERE (`users`.`id` = 18205844) 
30
+ line_definition :query_executed do |line|
31
+ line.regexp = /\s+(?:\e\[4;36;1m)?((?:\w+::)*\w+) Load \((\d+\.\d+)ms\)(?:\e\[0m)?\s+(?:\e\[0;1m)?(.+) (?:\e\[0m)?/
32
+ line.captures << { :name => :query_class, :type => :string } \
33
+ << { :name => :query_duration, :type => :msec } \
34
+ << { :name => :query_sql, :type => :string }
35
+ end
36
+
37
+ # CACHE (0.0ms) SELECT * FROM `users` WHERE (`users`.`id` = 0) 
38
+ line_definition :query_cached do |line|
39
+ line.teaser = /\s+(?:\e\[4;35;1m)?CACHE \((\d+\.\d+)ms\)(?:\e\[0m)?\s+(?:\e\[0m)?(.+) (?:\e\[0m)?/
40
+ line.regexp = /\s+(?:\e\[4;35;1m)?CACHE \((\d+\.\d+)ms\)(?:\e\[0m)?\s+(?:\e\[0m)?(.+) (?:\e\[0m)?/
41
+ line.captures << { :name => :cached_duration, :type => :msec } \
42
+ << { :name => :cached_sql, :type => :string }
43
+ end
44
+
45
+ # RuntimeError (Cannot destroy employee): /app/models/employee.rb:198:in `before_destroy'
46
+ line_definition :failed do |line|
47
+ line.footer = true
48
+ line.regexp = /((?:[A-Z]\w+\:\:)*[A-Z]\w+) \((.*)\)(?: on line #(\d+) of .+)?\:(.*)/
49
+ line.captures << { :name => :error, :type => :string } \
50
+ << { :name => :message, :type => :string } \
51
+ << { :name => :line, :type => :integer } \
52
+ << { :name => :file, :type => :string } \
53
+ << { :name => :stack_trace, :type => :string, :anonymize => true }
54
+ end
55
+
56
+
57
+ # Rails < 2.1 completed line example
58
+ # Completed in 0.21665 (4 reqs/sec) | Rendering: 0.00926 (4%) | DB: 0.00000 (0%) | 200 OK [http://demo.nu/employees]
59
+ RAILS_21_COMPLETED = /Completed in (\d+\.\d{5}) \(\d+ reqs\/sec\) (?:\| Rendering: (\d+\.\d{5}) \(\d+\%\) )?(?:\| DB: (\d+\.\d{5}) \(\d+\%\) )?\| (\d\d\d).+\[(http.+)\]/
60
+
61
+ # Rails > 2.1 completed line example
62
+ # Completed in 614ms (View: 120, DB: 31) | 200 OK [http://floorplanner.local/demo]
63
+ RAILS_22_COMPLETED = /Completed in (\d+)ms \((?:View: (\d+), )?DB: (\d+)\) \| (\d\d\d).+\[(http.+)\]/
64
+
65
+ # The completed line uses a kind of hack to ensure that both old style logs and new style logs
66
+ # are both parsed by the same regular expression. The format in Rails 2.2 was slightly changed,
67
+ # but the line contains exactly the same information.
68
+ line_definition :completed do |line|
69
+
70
+ line.footer = true
71
+ line.teaser = /Completed in /
72
+ line.regexp = Regexp.new("(?:#{RAILS_21_COMPLETED}|#{RAILS_22_COMPLETED})")
73
+
74
+ line.captures << { :name => :duration, :type => :sec, :anonymize => :slightly } \
75
+ << { :name => :view, :type => :sec, :anonymize => :slightly } \
76
+ << { :name => :db, :type => :sec, :anonymize => :slightly } \
77
+ << { :name => :status, :type => :integer } \
78
+ << { :name => :url, :type => :string, :anonymize => :url } # Old variant
79
+
80
+ line.captures << { :name => :duration, :type => :msec, :anonymize => :slightly } \
81
+ << { :name => :view, :type => :msec, :anonymize => :slightly } \
82
+ << { :name => :db, :type => :msec, :anonymize => :slightly } \
83
+ << { :name => :status, :type => :integer} \
84
+ << { :name => :url, :type => :string, :anonymize => :url } # 2.2 variant
85
+ end
86
+
87
+
88
+
89
+ REQUEST_CATEGORIZER = Proc.new do |request|
90
+ format = request[:format] || 'html'
91
+ "#{request[:controller]}##{request[:action]}.#{format} [#{request[:method]}]"
92
+ end
93
+
94
+ report do |analyze|
95
+ analyze.timespan :line_type => :processing
96
+ analyze.category :category => REQUEST_CATEGORIZER, :title => 'Top 20 hits', :amount => 20, :line_type => :processing
97
+ analyze.category :method, :title => 'HTTP methods'
98
+ analyze.category :status, :title => 'HTTP statuses returned'
99
+ analyze.category :category => lambda { |request| request =~ :cache_hit ? 'Cache hit' : 'No hit' }, :title => 'Rails action cache hits'
100
+
101
+ analyze.duration :duration, :category => REQUEST_CATEGORIZER, :title => "Request duration", :line_type => :completed
102
+ analyze.duration :view, :category => REQUEST_CATEGORIZER, :title => "Database time", :line_type => :completed
103
+ analyze.duration :db, :category => REQUEST_CATEGORIZER, :title => "View rendering time", :line_type => :completed
104
+
105
+ analyze.category :category => REQUEST_CATEGORIZER, :title => 'Process blockers (> 1 sec duration)',
106
+ :if => lambda { |request| request[:duration] && request[:duration] > 1.0 }, :amount => 20
107
+
108
+ analyze.hourly_spread :line_type => :processing
109
+ analyze.category :error, :title => 'Failed requests', :line_type => :failed, :amount => 20
110
+ end
111
+
112
+
113
+
114
+ end
@@ -73,4 +73,48 @@ describe RequestLogAnalyzer::LogParser, "Rails" do
73
73
  @log_parser.should_receive(:warn).with(:no_current_request, anything).twice
74
74
  @log_parser.parse_file(log_fixture(:rails_unordered))
75
75
  end
76
+ end
77
+
78
+ describe "RequestLogAnalyzer::FileFormat::RailsDevelopment - Rails with development details" do
79
+ include RequestLogAnalyzerSpecHelper
80
+
81
+ before(:each) do
82
+ @file_format = RequestLogAnalyzer::FileFormat.load(:rails_development)
83
+ end
84
+
85
+ it "should have a valid language definitions" do
86
+ @file_format.should be_valid
87
+ end
88
+
89
+ it "should parse a rendered line" do
90
+ info = @file_format.line_definitions[:rendered].matches("Rendered layouts/_footer (2.9ms)")
91
+ info[:render_file].should == 'layouts/_footer'
92
+ info[:render_duration].should == 0.0029
93
+ end
94
+
95
+ it "should parse a query executed line with colors" do
96
+ info = @file_format.line_definitions[:query_executed].matches(" User Load (0.4ms) SELECT * FROM `users` WHERE (`users`.`id` = 18205844) ")
97
+ info[:query_class].should == 'User'
98
+ info[:query_duration].should == 0.0004
99
+ info[:query_sql].should == 'SELECT * FROM `users` WHERE (`users`.`id` = 18205844)'
100
+ end
101
+
102
+ it "should parse a query executed line without colors" do
103
+ info = @file_format.line_definitions[:query_executed].matches(" User Load (0.4ms) SELECT * FROM `users` WHERE (`users`.`id` = 18205844) ")
104
+ info[:query_class].should == 'User'
105
+ info[:query_duration].should == 0.0004
106
+ info[:query_sql].should == 'SELECT * FROM `users` WHERE (`users`.`id` = 18205844)'
107
+ end
108
+
109
+ it "should parse a cached query line with colors" do
110
+ info = @file_format.line_definitions[:query_cached].matches(' CACHE (0.0ms) SELECT * FROM `users` WHERE (`users`.`id` = 0) ')
111
+ info[:cached_duration].should == 0.0
112
+ info[:cached_sql].should == 'SELECT * FROM `users` WHERE (`users`.`id` = 0)'
113
+ end
114
+
115
+ it "should parse a cached query line without colors" do
116
+ info = @file_format.line_definitions[:query_cached].matches(' CACHE (0.0ms) SELECT * FROM `users` WHERE (`users`.`id` = 0) ')
117
+ info[:cached_duration].should == 0.0
118
+ info[:cached_sql].should == 'SELECT * FROM `users` WHERE (`users`.`id` = 0)'
119
+ end
76
120
  end
@@ -2,7 +2,6 @@ require 'rubygems'
2
2
  require 'rubyforge'
3
3
  require 'rake'
4
4
  require 'rake/tasklib'
5
- require 'rake/gempackagetask'
6
5
  require 'date'
7
6
 
8
7
  module Rake
@@ -26,13 +25,56 @@ module Rake
26
25
  namespace(:gem) do
27
26
  desc "Updates the file lists for this gem"
28
27
  task(:manifest) { manifest_task }
28
+
29
+ desc "Releases a new version of #{@name}"
30
+ task(:build => [:manifest]) { build_task }
29
31
 
30
32
  desc "Releases a new version of #{@name}"
31
- task(:release => :package) { release_task }
33
+ task(:release => [:check_clean_master_branch, :version, :build]) { release_task }
34
+
35
+ # helper task for releasing
36
+ task(:check_clean_master_branch) { verify_clean_status('master') }
37
+ task(:check_version) { verify_version(ENV['VERSION'] || @specification.version) }
38
+ task(:version => [:check_version]) { set_gem_version! }
39
+ end
40
+
41
+ # Register RDoc tasks
42
+ if @specification.has_rdoc
43
+ require 'rake/rdoctask'
32
44
 
33
- Rake::GemPackageTask.new(@specification) do |pkg|
45
+ namespace(:doc) do
46
+ desc 'Generate documentation for request-log-analyzer'
47
+ Rake::RDocTask.new(:compile) do |rdoc|
48
+ rdoc.rdoc_dir = 'doc'
49
+ rdoc.title = @name
50
+ rdoc.options += @specification.rdoc_options
51
+ rdoc.rdoc_files.include(@specification.extra_rdoc_files)
52
+ rdoc.rdoc_files.include('lib/**/*.rb')
53
+ end
54
+ end
55
+ end
56
+
57
+ # Setup :spec task if RSpec files exist
58
+ if Dir['spec/**/*_spec.rb'].any?
59
+ require 'spec/rake/spectask'
60
+
61
+ desc "Run all specs for #{@name}"
62
+ Spec::Rake::SpecTask.new(:spec) do |t|
63
+ t.spec_files = FileList['spec/**/*_spec.rb']
64
+ end
65
+ end
66
+
67
+ # Setup :test task if unit test files exist
68
+ if Dir['test/**/*_test.rb'].any?
69
+ require 'rake/testtask'
70
+
71
+ desc "Run all unit tests for #{@name}"
72
+ Rake::TestTask.new(:test) do |t|
73
+ t.pattern = 'test/**/*_test.rb'
74
+ t.verbose = true
75
+ t.libs << 'test'
34
76
  end
35
- end
77
+ end
36
78
  end
37
79
 
38
80
  protected
@@ -110,6 +152,12 @@ module Rake
110
152
  newest_version = run_command('git tag').map { |tag| tag.split(name + '-').last }.compact.map { |v| Gem::Version.new(v) }.max
111
153
  raise "This version number (#{new_version}) is not higher than the highest tagged version (#{newest_version})" if !newest_version.nil? && newest_version >= Gem::Version.new(new_version.to_s)
112
154
  end
155
+
156
+ def set_gem_version!
157
+ # update gemspec file
158
+ self.gemspec_version = ENV['VERSION'] if Gem::Version.correct?(ENV['VERSION'])
159
+ self.gemspec_date = Date.today
160
+ end
113
161
 
114
162
  def manifest_task
115
163
  verify_current_branch('master')
@@ -139,26 +187,22 @@ module Rake
139
187
 
140
188
  def build_task
141
189
  sh "gem build #{gemspec_file}"
190
+ Dir.mkdir('pkg') unless File.exist?('pkg')
191
+ sh "mv #{name}-#{specification.version}.gem pkg/#{name}-#{specification.version}.gem"
142
192
  end
143
193
 
144
194
  def install_task
145
- raise "#{name} .gem file not found" unless File.exist?("#{name}-#{specification.version}.gem")
146
- sh "gem install #{name}-#{specification.version}.gem"
195
+ raise "#{name} .gem file not found" unless File.exist?("pkg/#{name}-#{specification.version}.gem")
196
+ sh "gem install pkg/#{name}-#{specification.version}.gem"
147
197
  end
148
198
 
149
199
  def uninstall_task
150
- raise "#{name} .gem file not found" unless File.exist?("#{name}-#{specification.version}.gem")
200
+ raise "#{name} .gem file not found" unless File.exist?("pkg/#{name}-#{specification.version}.gem")
151
201
  sh "gem uninstall #{name}"
152
202
  end
153
203
 
154
204
  def release_task
155
- verify_clean_status('master')
156
- verify_version(ENV['VERSION'] || @specification.version)
157
-
158
- # update gemspec file
159
- self.gemspec_version = ENV['VERSION'] if Gem::Version.correct?(ENV['VERSION'])
160
- self.gemspec_date = Date.today
161
- manifest_task
205
+ # commit the gemspec file
162
206
  git_commit_file(gemspec_file, "Updated #{gemspec_file} for release of version #{@specification.version}") if git_modified?(gemspec_file)
163
207
 
164
208
  # create tag and push changes
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: wvanbergen-request-log-analyzer
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.0.3
4
+ version: 1.0.4
5
5
  platform: ruby
6
6
  authors:
7
7
  - Willem van Bergen
@@ -10,7 +10,7 @@ autorequire:
10
10
  bindir: bin
11
11
  cert_chain: []
12
12
 
13
- date: 2009-01-14 00:00:00 -08:00
13
+ date: 2009-01-17 00:00:00 -08:00
14
14
  default_executable: request-log-analyzer
15
15
  dependencies: []
16
16
 
@@ -20,13 +20,13 @@ executables:
20
20
  - request-log-analyzer
21
21
  extensions: []
22
22
 
23
- extra_rdoc_files: []
24
-
23
+ extra_rdoc_files:
24
+ - README.rdoc
25
25
  files:
26
26
  - DESIGN
27
27
  - HACKING
28
28
  - LICENSE
29
- - README.textile
29
+ - README.rdoc
30
30
  - Rakefile
31
31
  - bin
32
32
  - bin/request-log-analyzer
@@ -47,6 +47,7 @@ files:
47
47
  - lib/request_log_analyzer/file_format.rb
48
48
  - lib/request_log_analyzer/file_format/merb.rb
49
49
  - lib/request_log_analyzer/file_format/rails.rb
50
+ - lib/request_log_analyzer/file_format/rails_development.rb
50
51
  - lib/request_log_analyzer/filter
51
52
  - lib/request_log_analyzer/filter/anonimize.rb
52
53
  - lib/request_log_analyzer/filter/base.rb
@@ -95,12 +96,16 @@ files:
95
96
  - tasks
96
97
  - tasks/github-gem.rake
97
98
  - tasks/request_log_analyzer.rake
98
- - tasks/rspec.rake
99
- has_rdoc: false
99
+ has_rdoc: true
100
100
  homepage: http://github.com/wvanbergen/request-log-analyzer/wikis
101
101
  post_install_message:
102
- rdoc_options: []
103
-
102
+ rdoc_options:
103
+ - --title
104
+ - request-log-analyzer
105
+ - --main
106
+ - README.rdoc
107
+ - --line-numbers
108
+ - --inline-source
104
109
  require_paths:
105
110
  - lib
106
111
  required_ruby_version: !ruby/object:Gem::Requirement
data/README.textile DELETED
@@ -1,36 +0,0 @@
1
- h1. Request-log-analyzer
2
-
3
- This is a simple command line tool to analyze request log files of both Rails and
4
- Merb to produce a performance report. Its purpose is to find what actions are best candidates for optimization.
5
-
6
- * Analyzes Rails log files (all versions)
7
- * Can combine multiple files (handy if you are using logrotate)
8
- * Uses several metrics, including cumulative request time, average request time, process blockers, database and rendering time, HTTP methods and states, Rails action cache statistics, etc.) ("Sample output":http://wiki.github.com/wvanbergen/request-log-analyzer/sample-output)
9
- * Low memory footprint (server-safe)
10
- * Fast
11
- * MIT licensed
12
-
13
- h2. Installation
14
-
15
- <pre>
16
- $ sudo gem install wvanbergen-request-log-analyzer --source http://gems.github.com
17
- </pre>
18
-
19
- To get the best results out of request-log-analyzer, make sure to
20
- "set up logging correctly":http://wiki.github.com/wvanbergen/request-log-analyzer/configure-logging
21
- for your application.
22
-
23
- h2. Usage
24
-
25
- To analyze a log file and produce a performance report, run request-log-analyzer like this:
26
-
27
- <pre>
28
- $ request-log-analyzer log/production.log
29
- </pre>
30
-
31
- For more details and available command line options, see the "project's wiki":http://wiki.github.com/wvanbergen/request-log-analyzer/basic-usage
32
-
33
- h2. Additional information
34
-
35
- * "Project wiki at GitHub":http://wiki.github.com/wvanbergen/request-log-analyzer
36
- * "wvanbergen's blog posts":http://techblog.floorplanner.com/tag/request-log-analyzer/
data/tasks/rspec.rake DELETED
@@ -1,6 +0,0 @@
1
- require 'spec/rake/spectask'
2
-
3
- desc "Run all specs in spec directory (excluding plugin specs)"
4
- Spec::Rake::SpecTask.new(:spec) do |t|
5
- t.spec_files = FileList['spec/**/*_spec.rb']
6
- end