wvanbergen-request-log-analyzer 1.1.2 → 1.1.3

Sign up to get free protection for your applications and to get access to all the features.
Files changed (52) hide show
  1. data/DESIGN +24 -10
  2. data/bin/request-log-analyzer +2 -27
  3. data/lib/cli/progressbar.rb +2 -19
  4. data/lib/cli/tools.rb +46 -0
  5. data/lib/request_log_analyzer/aggregator/database.rb +16 -5
  6. data/lib/request_log_analyzer/aggregator/echo.rb +1 -0
  7. data/lib/request_log_analyzer/aggregator/summarizer.rb +15 -13
  8. data/lib/request_log_analyzer/controller.rb +8 -4
  9. data/lib/request_log_analyzer/file_format/merb.rb +17 -4
  10. data/lib/request_log_analyzer/file_format/rails_development.rb +30 -92
  11. data/lib/request_log_analyzer/file_format.rb +8 -4
  12. data/lib/request_log_analyzer/filter/anonymize.rb +0 -3
  13. data/lib/request_log_analyzer/filter/field.rb +6 -1
  14. data/lib/request_log_analyzer/filter/timespan.rb +7 -1
  15. data/lib/request_log_analyzer/filter.rb +0 -4
  16. data/lib/request_log_analyzer/line_definition.rb +13 -2
  17. data/lib/request_log_analyzer/output/fixed_width.rb +7 -1
  18. data/lib/request_log_analyzer/output/html.rb +1 -0
  19. data/lib/request_log_analyzer/request.rb +4 -0
  20. data/lib/request_log_analyzer/source/log_parser.rb +19 -21
  21. data/lib/request_log_analyzer/source.rb +2 -1
  22. data/lib/request_log_analyzer/tracker/duration.rb +74 -14
  23. data/lib/request_log_analyzer/tracker/frequency.rb +22 -10
  24. data/lib/request_log_analyzer/tracker/hourly_spread.rb +18 -6
  25. data/lib/request_log_analyzer/tracker/timespan.rb +15 -8
  26. data/lib/request_log_analyzer.rb +4 -8
  27. data/spec/integration/command_line_usage_spec.rb +71 -0
  28. data/spec/lib/helper.rb +33 -0
  29. data/spec/lib/mocks.rb +47 -0
  30. data/spec/{file_formats/spec_format.rb → lib/testing_format.rb} +6 -1
  31. data/spec/spec_helper.rb +5 -41
  32. data/spec/{database_inserter_spec.rb → unit/aggregator/database_inserter_spec.rb} +40 -37
  33. data/spec/unit/aggregator/summarizer_spec.rb +28 -0
  34. data/spec/unit/controller/controller_spec.rb +43 -0
  35. data/spec/{log_processor_spec.rb → unit/controller/log_processor_spec.rb} +4 -3
  36. data/spec/{file_format_spec.rb → unit/file_format/file_format_api_spec.rb} +16 -4
  37. data/spec/{line_definition_spec.rb → unit/file_format/line_definition_spec.rb} +13 -6
  38. data/spec/{merb_format_spec.rb → unit/file_format/merb_format_spec.rb} +2 -2
  39. data/spec/{rails_format_spec.rb → unit/file_format/rails_format_spec.rb} +19 -11
  40. data/spec/unit/filter/anonymize_filter_spec.rb +22 -0
  41. data/spec/unit/filter/field_filter_spec.rb +69 -0
  42. data/spec/unit/filter/timespan_filter_spec.rb +61 -0
  43. data/spec/{log_parser_spec.rb → unit/source/log_parser_spec.rb} +7 -7
  44. data/spec/{request_spec.rb → unit/source/request_spec.rb} +5 -5
  45. data/spec/unit/tracker/duration_tracker_spec.rb +90 -0
  46. data/spec/unit/tracker/frequency_tracker_spec.rb +83 -0
  47. data/spec/unit/tracker/timespan_tracker_spec.rb +64 -0
  48. data/spec/unit/tracker/tracker_api_test.rb +45 -0
  49. metadata +50 -26
  50. data/spec/controller_spec.rb +0 -64
  51. data/spec/filter_spec.rb +0 -157
  52. data/spec/summarizer_spec.rb +0 -9
data/DESIGN CHANGED
@@ -4,21 +4,35 @@ This allows you to easily add extra reports, filters and outputs.
4
4
 
5
5
  1) Build pipeline.
6
6
  -> Aggregator (database)
7
- Source -> Filter -> Filter -> Aggregator (summary report)
7
+ Source -> Filter -> Filter -> Aggregator (summary report) -> Output
8
8
  -> Aggregator (...)
9
-
9
+
10
10
  2) Start chunk producer and push chunks through pipeline.
11
- Controller.start
11
+ Controller.start
12
+
13
+ RequestLogAnalyzer::Source is an Object that pushes requests into the chain.
14
+ At the moment you can only use the log-parser as a source.
15
+ It accepts files or stdin and can parse then into request objects using a RequestLogAnalyzer::FileFormat definition.
16
+ In the future we want to be able to have a generated request database as source as this will make interactive
17
+ down drilling possible.
18
+
19
+ The filters are all subclasses of the RequestLogAnalyzer::Filter class.
20
+ They accept a request object, manipulate or drop it, and then pass the request object on to the next filter
21
+ in the chain.
22
+ At the moment there are three types of filters available: Anonymize, Field and Timespan.
23
+
24
+ The Aggregators all inherit from the RequestLogAnalyzer::Aggregator class.
25
+ All the requests that come out of the Filterchain are fed into all the aggregators in parallel.
26
+ These aggregators can do anything what they want with the given request.
27
+ For example: the Database aggregator will just store all the requests into a SQLite database while the Summarizer will
28
+ generate a wide range of statistical reports from them.
12
29
 
13
30
  3) Gather output from pipeline.
14
31
  Controller.report
15
32
 
16
- At the moment the supported sources are file and STDIN.
17
- In the future we want to be able to have a generated request database as source.
18
- This will make interactive downdrilling possible.
19
-
33
+ All Aggregators are asked to report what they have done. For example the database will report: I stuffed x requests
34
+ into SQLite database Y. The Summarizer will output its reports.
20
35
 
21
- For the report generation output we now use the File and the STDOUT class, as they both support <<.
22
- In the future we want to have a OutputFile, OutputSTDOUT and OutputHTML class, so that reports can generate
23
- tables, lines and comments and push them into the output class.
36
+ The output is pushed to a RequestLogAnalyzer::Output object, which takes care of the output.
37
+ It can generate either ASCII, UTF8 or even HTML output.
24
38
 
@@ -1,15 +1,7 @@
1
1
  #!/usr/bin/ruby
2
2
  require File.dirname(__FILE__) + '/../lib/request_log_analyzer'
3
3
  require File.dirname(__FILE__) + '/../lib/cli/command_line_arguments'
4
-
5
- def terminal_width(default = 81)
6
- IO.popen('stty -a') do |pipe|
7
- column_line = pipe.detect { |line| /(\d+) columns/ =~ line }
8
- width = column_line ? $1.to_i : default
9
- end
10
- rescue
11
- default
12
- end
4
+ require File.dirname(__FILE__) + '/../lib/cli/tools'
13
5
 
14
6
  # Parse the arguments given via commandline
15
7
  begin
@@ -29,7 +21,7 @@ begin
29
21
 
30
22
  command_line.option(:format, :alias => :f, :default => 'rails')
31
23
  command_line.option(:file, :alias => :e)
32
- command_line.switch(:assume_correct_order)
24
+ command_line.option(:parse_strategy, :default => 'assume-correct')
33
25
 
34
26
  command_line.option(:aggregator, :alias => :a, :multiple => true)
35
27
  command_line.option(:database, :alias => :d)
@@ -80,23 +72,6 @@ rescue CommandLine::Error => e
80
72
  exit(0)
81
73
  end
82
74
 
83
- def install_rake_tasks(install_type)
84
- if install_type == 'rails'
85
- require 'ftools'
86
- if File.directory?('./lib/tasks/')
87
- File.copy(File.dirname(__FILE__) + '/../tasks/request_log_analyzer.rake', './lib/tasks/request_log_analyze.rake')
88
- puts "Installed rake tasks."
89
- puts "To use, run: rake log:analyze"
90
- else
91
- puts "Cannot find /lib/tasks folder. Are you in your Rails directory?"
92
- puts "Installation aborted."
93
- end
94
- else
95
- raise "Cannot perform this install type! (#{install_type})"
96
- end
97
- end
98
-
99
-
100
75
  case arguments.command
101
76
  when :install
102
77
  install_rake_tasks(arguments.parameters[0])
@@ -119,23 +119,6 @@ module CommandLine
119
119
  end
120
120
  end
121
121
 
122
- def get_width
123
- # FIXME: I don't know how portable it is.
124
- default_width = 80
125
- begin
126
- tiocgwinsz = 0x5413
127
- data = [0, 0, 0, 0].pack("SSSS")
128
- if @out.ioctl(tiocgwinsz, data) >= 0 then
129
- rows, cols, xpixels, ypixels = data.unpack("SSSS")
130
- if cols >= 0 then cols else default_width end
131
- else
132
- default_width
133
- end
134
- rescue Exception
135
- default_width
136
- end
137
- end
138
-
139
122
  def show
140
123
  arguments = @format_arguments.map {|method|
141
124
  method = sprintf("fmt_%s", method)
@@ -143,7 +126,7 @@ module CommandLine
143
126
  }
144
127
  line = sprintf(@format, *arguments)
145
128
 
146
- width = get_width
129
+ width = terminal_width(80)
147
130
  if line.length == width - 1
148
131
  @out.print(line + eol)
149
132
  @out.flush
@@ -176,7 +159,7 @@ module CommandLine
176
159
  public
177
160
  def clear
178
161
  @out.print "\r"
179
- @out.print(" " * (get_width - 1))
162
+ @out.print(" " * (terminal_width(80) - 1))
180
163
  @out.print "\r"
181
164
  end
182
165
 
data/lib/cli/tools.rb ADDED
@@ -0,0 +1,46 @@
1
+ # Try to determine the terminal with.
2
+ # If it is not possible to to so, it returns the default_width.
3
+ # <tt>default_width</tt> Defaults to 81
4
+ def terminal_width(default_width = 81)
5
+ tiocgwinsz = 0x5413
6
+ data = [0, 0, 0, 0].pack("SSSS")
7
+ if @out.ioctl(tiocgwinsz, data) >= 0
8
+ rows, cols, xpixels, ypixels = data.unpack("SSSS")
9
+ raise unless cols > 0
10
+ cols
11
+ else
12
+ raise
13
+ end
14
+ rescue
15
+ begin
16
+ IO.popen('stty -a') do |pipe|
17
+ column_line = pipe.detect { |line| /(\d+) columns/ =~ line }
18
+ raise unless column_line
19
+ $1.to_i
20
+ end
21
+ rescue
22
+ default_width
23
+ end
24
+ end
25
+
26
+ # Copies request-log-analyzer analyzer rake tasks into the /lib/tasks folder of a project, for easy access and
27
+ # environment integration.
28
+ # <tt>install_type</tt> Type of project to install into. Defaults to :rails.
29
+ # Raises if it cannot find the project folder or if the install_type is now known.
30
+ def install_rake_tasks(install_type = :rails)
31
+ if install_type.to_sym == :rails
32
+ require 'ftools'
33
+ if File.directory?('./lib/tasks/')
34
+ File.copy(File.dirname(__FILE__) + '/../tasks/request_log_analyzer.rake', './lib/tasks/request_log_analyze.rake')
35
+ puts "Installed rake tasks."
36
+ puts "To use, run: rake log:analyze"
37
+ else
38
+ puts "Cannot find /lib/tasks folder. Are you in your Rails directory?"
39
+ puts "Installation aborted."
40
+ end
41
+ else
42
+ raise "Cannot perform this install type! (#{install_type.to_s})"
43
+ end
44
+ end
45
+
46
+
@@ -30,7 +30,8 @@ module RequestLogAnalyzer::Aggregator
30
30
  def aggregate(request)
31
31
  @request_object = @request_class.new(:first_lineno => request.first_lineno, :last_lineno => request.last_lineno)
32
32
  request.lines.each do |line|
33
- attributes = line.reject { |k, v| [:line_type].include?(k) }
33
+ class_columns = @orm_module.const_get("#{line[:line_type]}_line".classify).column_names.reject { |column| ['id'].include?(column) }
34
+ attributes = Hash[*line.select { |(k, v)| class_columns.include?(k.to_s) }.flatten]
34
35
  @request_object.send("#{line[:line_type]}_lines").build(attributes)
35
36
  end
36
37
  @request_object.save!
@@ -72,7 +73,16 @@ module RequestLogAnalyzer::Aggregator
72
73
  t.column(:request_id, :integer)
73
74
  t.column(:lineno, :integer)
74
75
  definition.captures.each do |capture|
75
- t.column(capture[:name], column_type(capture))
76
+
77
+ # Add a field for every capture
78
+ t.column(capture[:name], column_type(capture[:type]))
79
+
80
+ # If the capture provides other field as well, create them too
81
+ if capture[:provides].kind_of?(Hash)
82
+ capture[:provides].each do |field, field_type|
83
+ t.column(field, column_type(field_type))
84
+ end
85
+ end
76
86
  end
77
87
  end
78
88
  ActiveRecord::Migration.add_index("#{name}_lines", [:request_id])
@@ -136,12 +146,13 @@ module RequestLogAnalyzer::Aggregator
136
146
 
137
147
  # Function to determine the column type for a field
138
148
  # TODO: make more robust / include in file-format definition
139
- def column_type(capture)
140
- case capture[:type]
149
+ def column_type(type_indicator)
150
+ case type_indicator
151
+ when :eval; :text
141
152
  when :sec; :double
142
153
  when :msec; :double
143
154
  when :float; :double
144
- else capture[:type]
155
+ else type_indicator
145
156
  end
146
157
  end
147
158
  end
@@ -1,5 +1,6 @@
1
1
  module RequestLogAnalyzer::Aggregator
2
2
 
3
+ # Echo Aggregator. Writes everything passed to it
3
4
  class Echo < Base
4
5
 
5
6
  def prepare
@@ -1,5 +1,3 @@
1
- require File.dirname(__FILE__) + '/../tracker'
2
-
3
1
  module RequestLogAnalyzer::Aggregator
4
2
 
5
3
  class Summarizer < Base
@@ -11,6 +9,10 @@ module RequestLogAnalyzer::Aggregator
11
9
  def initialize
12
10
  @trackers = []
13
11
  end
12
+
13
+ def initialize_copy(other)
14
+ @trackers = other.trackers.dup
15
+ end
14
16
 
15
17
  def reset!
16
18
  @trackers = []
@@ -58,7 +60,7 @@ module RequestLogAnalyzer::Aggregator
58
60
  def prepare
59
61
  raise "No trackers set up in Summarizer!" if @trackers.nil? || @trackers.empty?
60
62
  @trackers.each { |tracker| tracker.prepare }
61
- end
63
+ end
62
64
 
63
65
  def aggregate(request)
64
66
  @trackers.each do |tracker|
@@ -86,23 +88,23 @@ module RequestLogAnalyzer::Aggregator
86
88
 
87
89
  output.with_style(:cell_separator => false) do
88
90
  output.table({:width => 20}, {:font => :bold}) do |rows|
89
- rows << ['Parsed lines:', source.parsed_lines]
90
- rows << ['Parsed request:', source.parsed_requests]
91
- rows << ['Skipped lines:', source.skipped_lines]
91
+ rows << ['Parsed lines:', source.parsed_lines]
92
+ rows << ['Parsed requests:', source.parsed_requests]
93
+ rows << ['Skipped lines:', source.skipped_lines]
92
94
 
93
- rows << ["Warnings:", @warnings_encountered.map { |(key, value)| "#{key.inspect}: #{value}" }.join(', ')] if has_warnings?
95
+ rows << ["Warnings:", @warnings_encountered.map { |(key, value)| "#{key}: #{value}" }.join(', ')] if has_warnings?
94
96
  end
95
97
  end
96
98
  output << "\n"
97
99
  end
98
100
 
99
101
  def report_footer(output)
100
- if has_serious_warnings?
101
-
102
+ if has_log_ordering_warnings?
102
103
  output.title("Parse warnings")
103
104
 
104
- output.puts "Multiple warnings were encountered during parsing. Possibly, your logging "
105
- output.puts "is not setup correctly. Visit this website for logging configuration tips:"
105
+ output.puts "Parseable lines were ancountered without a header line before it. It"
106
+ output.puts "could be that logging is not setup correctly for your application."
107
+ output.puts "Visit this website for logging configuration tips:"
106
108
  output.puts output.link("http://github.com/wvanbergen/request-log-analyzer/wikis/configure-logging")
107
109
  output.puts
108
110
  end
@@ -112,8 +114,8 @@ module RequestLogAnalyzer::Aggregator
112
114
  @warnings_encountered.inject(0) { |result, (key, value)| result += value } > 0
113
115
  end
114
116
 
115
- def has_serious_warnings?
116
- @warnings_encountered.inject(0) { |result, (key, value)| result += value } > 10
117
+ def has_log_ordering_warnings?
118
+ @warnings_encountered[:no_current_request] && @warnings_encountered[:no_current_request] > 0
117
119
  end
118
120
 
119
121
  def warning(type, message, lineno)
@@ -64,7 +64,7 @@ module RequestLogAnalyzer
64
64
 
65
65
  controller = Controller.new(RequestLogAnalyzer::Source::LogParser.new(file_format, options), options)
66
66
 
67
- options[:assume_correct_order] = arguments[:assume_correct_order]
67
+ options[:parse_strategy] = arguments[:parse_strategy]
68
68
 
69
69
  # register filters
70
70
  if arguments[:after] || arguments[:before]
@@ -160,6 +160,9 @@ module RequestLogAnalyzer
160
160
  @filters << filter.new(file_format, @options.merge(filter_options))
161
161
  end
162
162
 
163
+ # Push a request through the entire filterchain (@filters).
164
+ # <tt>request</tt> The request to filter.
165
+ # Returns the filtered request or nil.
163
166
  def filter_request(request)
164
167
  @filters.each do |filter|
165
168
  request = filter.filter(request)
@@ -168,7 +171,10 @@ module RequestLogAnalyzer
168
171
  return request
169
172
  end
170
173
 
174
+ # Push a request to all the aggregators (@aggregators).
175
+ # <tt>request</tt> The request to push to the aggregators.
171
176
  def aggregate_request(request)
177
+ return unless request
172
178
  @aggregators.each { |agg| agg.aggregate(request) }
173
179
  end
174
180
 
@@ -182,13 +188,11 @@ module RequestLogAnalyzer
182
188
  # 6. Finalize Source
183
189
  def run!
184
190
 
185
- @filters.each { |filter| filter.prepare }
186
191
  @aggregators.each { |agg| agg.prepare }
187
192
 
188
193
  begin
189
194
  @source.each_request do |request|
190
- request = filter_request(request)
191
- aggregate_request(request) unless request.nil?
195
+ aggregate_request(filter_request(request))
192
196
  end
193
197
  rescue Interrupt => e
194
198
  handle_progress(:interrupted)
@@ -14,8 +14,9 @@ module RequestLogAnalyzer::FileFormat
14
14
  # ~ Params: {"_method"=>"delete", "authenticity_token"=>"[FILTERED]", "action"=>"d}
15
15
  line_definition :params do |line|
16
16
  line.teaser = /Params/
17
- line.regexp = /Params\:\ \{(.+)\}/
18
- line.captures << { :name => :raw_hash, :type => :string}
17
+ line.regexp = /Params\:\ (\{.+\})/
18
+ line.captures << { :name => :params, :type => :eval, :provides => {
19
+ :namespace => :string, :controller => :string, :action => :string, :format => :string } }
19
20
  end
20
21
 
21
22
  # ~ {:dispatch_time=>0.006117, :after_filters_time=>6.1e-05, :before_filters_time=>0.000712, :action_time=>0.005833}
@@ -29,11 +30,23 @@ module RequestLogAnalyzer::FileFormat
29
30
  << { :name => :action_time, :type => :duration }
30
31
  end
31
32
 
33
+ REQUEST_CATEGORIZER = Proc.new do |request|
34
+ category = "#{request[:controller]}##{request[:action]}"
35
+ category = "#{request[:namespace]}::#{category}" if request[:namespace]
36
+ category = "#{category}.#{request[:format]}" if request[:format]
37
+ category
38
+ end
32
39
 
33
40
  report do |analyze|
34
- # FIX ME
41
+ analyze.timespan :line_type => :started
42
+ analyze.hourly_spread :line_type => :started
43
+
44
+ analyze.duration :dispatch_time, :category => REQUEST_CATEGORIZER, :title => 'Request dispatch duration'
45
+ # analyze.duration :action_time, :category => REQUEST_CATEGORIZER, :title => 'Request action duration'
46
+ # analyze.duration :after_filters_time, :category => REQUEST_CATEGORIZER, :title => 'Request after_filter duration'
47
+ # analyze.duration :before_filters_time, :category => REQUEST_CATEGORIZER, :title => 'Request before_filter duration'
35
48
  end
36
-
49
+
37
50
  end
38
51
 
39
52
  end
@@ -1,28 +1,19 @@
1
1
  module RequestLogAnalyzer::FileFormat
2
2
 
3
- class RailsDevelopment < Base
3
+ # The RailsDevelopment FileFormat is an extention to the default Rails file format. It includes
4
+ # all lines of the normal Rails file format, but parses SQL queries and partial rendering lines
5
+ # as well.
6
+ class RailsDevelopment < Rails
4
7
 
5
- # Processing EmployeeController#index (for 123.123.123.123 at 2008-07-13 06:00:00) [GET]
6
- line_definition :processing do |line|
7
- line.header = true # this line is the first log line for a request
8
- line.teaser = /Processing /
9
- line.regexp = /Processing ((?:\w+::)?\w+)#(\w+)(?: to (\w+))? \(for (\d+\.\d+\.\d+\.\d+) at (\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d)\) \[([A-Z]+)\]/
10
- line.captures << { :name => :controller, :type => :string } \
11
- << { :name => :action, :type => :string } \
12
- << { :name => :format, :type => :string } \
13
- << { :name => :ip, :type => :string } \
14
- << { :name => :timestamp, :type => :timestamp } \
15
- << { :name => :method, :type => :string }
16
- end
17
-
18
- # Filter chain halted as [#<ActionController::Caching::Actions::ActionCacheFilter:0x2a999ad620 @check=nil, @options={:store_options=>{}, :layout=>nil, :cache_path=>#<Proc:0x0000002a999b8890@/app/controllers/cached_controller.rb:8>}>] rendered_or_redirected.
19
- line_definition :cache_hit do |line|
20
- line.regexp = /Filter chain halted as \[\#<ActionController::Caching::Actions::ActionCacheFilter:.+>\] rendered_or_redirected/
8
+ # Parameters: {"action"=>"demo", "controller"=>"page"}
9
+ line_definition :parameters do |line|
10
+ line.teaser = /Parameters/
11
+ line.regexp = /\s+Parameters:\s+(\{.*\})/
12
+ line.captures << { :name => :params, :type => :eval }
21
13
  end
22
14
 
23
15
  # Rendered layouts/_footer (2.9ms)
24
16
  line_definition :rendered do |line|
25
- line.teaser = /Rendered /
26
17
  line.regexp = /Rendered (\w+(?:\/\w+)+) \((\d+\.\d+)ms\)/
27
18
  line.captures << { :name => :render_file, :type => :string } \
28
19
  << { :name => :render_duration, :type => :duration, :unit => :msec }
@@ -30,90 +21,37 @@ module RequestLogAnalyzer::FileFormat
30
21
 
31
22
  # User Load (0.4ms) SELECT * FROM `users` WHERE (`users`.`id` = 18205844) 
32
23
  line_definition :query_executed do |line|
33
- line.regexp = /\s+(?:\e\[4;36;1m)?((?:\w+::)*\w+) Load \((\d+\.\d+)ms\)(?:\e\[0m)?\s+(?:\e\[0;1m)?(.+) (?:\e\[0m)?/
24
+ line.regexp = /\s+(?:\e\[4;36;1m)?((?:\w+::)*\w+) Load \((\d+\.\d+)ms\)(?:\e\[0m)?\s+(?:\e\[0;1m)?([^\e]+) ?(?:\e\[0m)?/
34
25
  line.captures << { :name => :query_class, :type => :string } \
35
26
  << { :name => :query_duration, :type => :duration, :unit => :msec } \
36
- << { :name => :query_sql, :type => :string }
27
+ << { :name => :query_sql, :type => :sql }
37
28
  end
38
29
 
39
30
  # CACHE (0.0ms) SELECT * FROM `users` WHERE (`users`.`id` = 0) 
40
31
  line_definition :query_cached do |line|
41
- line.teaser = /\s+(?:\e\[4;35;1m)?CACHE \((\d+\.\d+)ms\)(?:\e\[0m)?\s+(?:\e\[0m)?(.+) (?:\e\[0m)?/
42
- line.regexp = /\s+(?:\e\[4;35;1m)?CACHE \((\d+\.\d+)ms\)(?:\e\[0m)?\s+(?:\e\[0m)?(.+) (?:\e\[0m)?/
32
+ line.regexp = /\s+(?:\e\[4;35;1m)?CACHE \((\d+\.\d+)ms\)(?:\e\[0m)?\s+(?:\e\[0m)?([^\e]+) ?(?:\e\[0m)?/
43
33
  line.captures << { :name => :cached_duration, :type => :duration, :unit => :msec } \
44
- << { :name => :cached_sql, :type => :string }
34
+ << { :name => :cached_sql, :type => :sql }
45
35
  end
46
-
47
- # RuntimeError (Cannot destroy employee): /app/models/employee.rb:198:in `before_destroy'
48
- line_definition :failed do |line|
49
- line.footer = true
50
- line.regexp = /((?:[A-Z]\w+\:\:)*[A-Z]\w+) \((.*)\)(?: on line #(\d+) of .+)?\:(.*)/
51
- line.captures << { :name => :error, :type => :string } \
52
- << { :name => :message, :type => :string } \
53
- << { :name => :line, :type => :integer } \
54
- << { :name => :file, :type => :string } \
55
- << { :name => :stack_trace, :type => :string }
56
- end
57
-
58
36
 
59
- # Rails < 2.1 completed line example
60
- # Completed in 0.21665 (4 reqs/sec) | Rendering: 0.00926 (4%) | DB: 0.00000 (0%) | 200 OK [http://demo.nu/employees]
61
- RAILS_21_COMPLETED = /Completed in (\d+\.\d{5}) \(\d+ reqs\/sec\) (?:\| Rendering: (\d+\.\d{5}) \(\d+\%\) )?(?:\| DB: (\d+\.\d{5}) \(\d+\%\) )?\| (\d\d\d).+\[(http.+)\]/
62
-
63
- # Rails > 2.1 completed line example
64
- # Completed in 614ms (View: 120, DB: 31) | 200 OK [http://floorplanner.local/demo]
65
- RAILS_22_COMPLETED = /Completed in (\d+)ms \((?:View: (\d+), )?DB: (\d+)\) \| (\d\d\d).+\[(http.+)\]/
66
-
67
- # The completed line uses a kind of hack to ensure that both old style logs and new style logs
68
- # are both parsed by the same regular expression. The format in Rails 2.2 was slightly changed,
69
- # but the line contains exactly the same information.
70
- line_definition :completed do |line|
71
-
72
- line.footer = true
73
- line.teaser = /Completed in /
74
- line.regexp = Regexp.new("(?:#{RAILS_21_COMPLETED}|#{RAILS_22_COMPLETED})")
75
-
76
- line.captures << { :name => :duration, :type => :duration } \
77
- << { :name => :view, :type => :duration } \
78
- << { :name => :db, :type => :duration } \
79
- << { :name => :status, :type => :integer } \
80
- << { :name => :url, :type => :string } # Old variant
81
-
82
- line.captures << { :name => :duration, :type => :duration, :unit => :msec } \
83
- << { :name => :view, :type => :duration, :unit => :msec } \
84
- << { :name => :db, :type => :duration, :unit => :msec } \
85
- << { :name => :status, :type => :integer} \
86
- << { :name => :url, :type => :string } # 2.2 variant
87
- end
88
-
89
- REQUEST_CATEGORIZER = Proc.new do |request|
90
- format = request[:format] || 'html'
91
- "#{request[:controller]}##{request[:action]}.#{format} [#{request[:method]}]"
92
- end
93
-
94
- report do |analyze|
95
- analyze.timespan :line_type => :processing
96
- analyze.frequency :category => REQUEST_CATEGORIZER, :title => 'Top 20 hits', :amount => 20, :line_type => :processing
97
- analyze.frequency :method, :title => 'HTTP methods'
98
- analyze.frequency :status, :title => 'HTTP statuses returned'
99
- analyze.frequency :category => lambda { |request| request =~ :cache_hit ? 'Cache hit' : 'No hit' }, :title => 'Rails action cache hits'
100
-
101
- analyze.duration :duration, :category => REQUEST_CATEGORIZER, :title => "Request duration", :line_type => :completed
102
- analyze.duration :view, :category => REQUEST_CATEGORIZER, :title => "Database time", :line_type => :completed
103
- analyze.duration :db, :category => REQUEST_CATEGORIZER, :title => "View rendering time", :line_type => :completed
104
-
105
- analyze.frequency :category => REQUEST_CATEGORIZER, :title => 'Process blockers (> 1 sec duration)',
106
- :if => lambda { |request| request[:duration] && request[:duration] > 1.0 }, :amount => 20
107
-
108
- analyze.hourly_spread :line_type => :processing
109
- analyze.frequency :error, :title => 'Failed requests', :line_type => :failed, :amount => 20
110
- end
37
+ # Define the reporting for the additional parsed lines
38
+ report(:append) do |analyze|
39
+
40
+ analyze.duration :render_duration, :category => :render_file, :multiple_per_request => true,
41
+ :amount => 20, :title => 'Partial rendering duration'
42
+
43
+ analyze.duration :query_duration, :category => :query_sql, :multiple_per_request => true,
44
+ :amount => 20, :title => 'Query duration'
45
+
46
+ end
111
47
 
112
- class Request < RequestLogAnalyzer::Request
113
-
114
- def convert_timestamp(value, definition)
115
- value.gsub(/[^0-9]/)[0...14].to_i unless value.nil?
48
+ # Add a converter method for the SQL fields the the Rails request class
49
+ class Request < Rails::Request
50
+
51
+ # Sanitizes SQL queries so that they can be grouped
52
+ def convert_sql(sql, definition)
53
+ sql.gsub(/\b\d+\b/, ':int').gsub(/`([^`]+)`/, '\1').gsub(/'[^']*'/, ':string').rstrip
116
54
  end
117
- end
55
+ end
118
56
  end
119
57
  end
@@ -80,8 +80,8 @@ module RequestLogAnalyzer::FileFormat
80
80
 
81
81
  # Copy the line and report definer from the parent class.
82
82
  subclass.class_eval do
83
- instance_variable_set(:@line_definer, superclass.line_definer)
84
- instance_variable_set(:@report_definer, superclass.report_definer)
83
+ instance_variable_set(:@line_definer, superclass.line_definer.clone)
84
+ instance_variable_set(:@report_definer, superclass.report_definer.clone)
85
85
  class << self; attr_accessor :line_definer, :report_definer; end
86
86
  end
87
87
 
@@ -95,8 +95,12 @@ module RequestLogAnalyzer::FileFormat
95
95
  @line_definer.send(name, &block)
96
96
  end
97
97
 
98
- def create_request(*hashes)
99
- self.class::Request.create(self, *hashes)
98
+ def request_class
99
+ self.class::Request
100
+ end
101
+
102
+ def request(*hashes)
103
+ request_class.create(self, *hashes)
100
104
  end
101
105
 
102
106
  # Specifies multiple line definitions at once using a block
@@ -7,9 +7,6 @@ module RequestLogAnalyzer::Filter
7
7
  # * <tt>:value</tt> Value that the field should match to be accepted or rejected.
8
8
  class Anonymize < Base
9
9
 
10
- def prepare
11
- end
12
-
13
10
  def generate_random_ip
14
11
  "#{rand(256)}.#{rand(256)}.#{rand(256)}.#{rand(256)}"
15
12
  end
@@ -9,8 +9,13 @@ module RequestLogAnalyzer::Filter
9
9
 
10
10
  attr_reader :field, :value, :mode
11
11
 
12
+ def initialize(file_format, options = {})
13
+ super(file_format, options)
14
+ setup_filter
15
+ end
16
+
12
17
  # Setup mode, field and value.
13
- def prepare
18
+ def setup_filter
14
19
  @mode = (@options[:mode] || :accept).to_sym
15
20
  @field = @options[:field].to_sym
16
21
 
@@ -8,9 +8,15 @@ module RequestLogAnalyzer::Filter
8
8
 
9
9
  attr_reader :before, :after
10
10
 
11
+ def initialize(file_format, options = {})
12
+ super(file_format, options)
13
+ setup_filter
14
+ end
15
+
16
+
11
17
  # Convert the timestamp to the correct formats for quick timestamp comparisons.
12
18
  # These are stored in the before and after attr_reader fields.
13
- def prepare
19
+ def setup_filter
14
20
  @after = @options[:after].strftime('%Y%m%d%H%M%S').to_i if options[:after]
15
21
  @before = @options[:before].strftime('%Y%m%d%H%M%S').to_i if options[:before]
16
22
  end
@@ -23,10 +23,6 @@ module RequestLogAnalyzer::Filter
23
23
  register_file_format(format)
24
24
  end
25
25
 
26
- # Initialize the filter
27
- def prepare
28
- end
29
-
30
26
  # Return the request if the request should be kept.
31
27
  # Return nil otherwise.
32
28
  def filter(request)
@@ -12,6 +12,10 @@ module RequestLogAnalyzer
12
12
  def initialize
13
13
  @line_definitions = {}
14
14
  end
15
+
16
+ def initialize_copy(other)
17
+ @line_definitions = other.line_definitions.dup
18
+ end
15
19
 
16
20
  def method_missing(name, *args, &block)
17
21
  if block_given?
@@ -33,7 +37,7 @@ module RequestLogAnalyzer
33
37
  @captures = []
34
38
  definition.each { |key, value| self.send("#{key.to_s}=".to_sym, value) }
35
39
  end
36
-
40
+
37
41
  def self.define(name, &block)
38
42
  definition = self.new(name)
39
43
  yield(definition) if block_given?
@@ -73,7 +77,14 @@ module RequestLogAnalyzer
73
77
  def convert_captured_values(values, request)
74
78
  value_hash = {}
75
79
  captures.each_with_index do |capture, index|
76
- value_hash[capture[:name]] ||= request.convert_value(values[index], capture)
80
+ converted = request.convert_value(values[index], capture)
81
+ if converted.kind_of?(Hash)
82
+ value_hash[capture[:name]] = values[index]
83
+ converted = converted.inject({}) { |h, (key, value)| h[key.to_sym] = value; h }
84
+ value_hash = converted.merge(value_hash)
85
+ else
86
+ value_hash[capture[:name]] ||= converted
87
+ end
77
88
  end
78
89
  return value_hash
79
90
  end