oink 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
data/History.txt ADDED
@@ -0,0 +1,4 @@
1
+ === 0.1.0 / 2009-02-09
2
+
3
+ * 1st release
4
+
data/MIT-LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ Copyright (c) 2009 Noah Davis
2
+ Copyright (c) 2008 Ben Johnson of Binary Logic (binarylogic.com)
3
+
4
+ Permission is hereby granted, free of charge, to any person obtaining
5
+ a copy of this software and associated documentation files (the
6
+ "Software"), to deal in the Software without restriction, including
7
+ without limitation the rights to use, copy, modify, merge, publish,
8
+ distribute, sublicense, and/or sell copies of the Software, and to
9
+ permit persons to whom the Software is furnished to do so, subject to
10
+ the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be
13
+ included in all copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
16
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
17
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
18
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
19
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
20
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
21
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
data/README.rdoc ADDED
@@ -0,0 +1,136 @@
1
+ === Oink
2
+
3
+ * http://github.com/noahd1/oink
4
+
5
+ === Description
6
+
7
+ Rails plugin and log parser to help narrow down the source(s) of increased memory usage in rails applications.
8
+
9
+ === Synopsis
10
+
11
+ Oink adds memory and active record instantiation information to rails log during runtime and provides an executable to help digest the enhanced logs.
12
+
13
+ Given a minimum threshold and a metric (memory or active record instantiation), the oink executable reports:
14
+
15
+ 1. The top ten single requests which exceeded the threshold for the metric, ordered by the request which exceeded the threshold the most
16
+ 2. The number of times each action exceeded the threshold for the metric, ordered by the action which exceeded the threshold the most
17
+ 3. (in verbose mode) The log lines produced by requests which exceeded the threshold
18
+
19
+ Many thanks to Ben Johnson for Memory Usage Logger (http://github.com/binarylogic/memory_usage_logger/tree/master) which is the basis of this plugin.
20
+
21
+ == Dependencies
22
+
23
+ Currently oink can only parse logs in the Hodel3000Logger format
24
+
25
+ - http://github.com/topfunky/hodel_3000_compliant_logger
26
+
27
+ When used as a gem, this is automatically brought in as a dependency.
28
+
29
+ === Installation and configuration
30
+
31
+ To begin, you'll want to install it:
32
+
33
+ gem install oink
34
+
35
+ Next, you'll need to update your config/environment.rb to include oink as a dependency, and to configure Rails to use the hodel_3000_compliant_logger:
36
+
37
+ Rails::Initializer.run do |config|
38
+ # your other config here
39
+ config.gem 'oink'
40
+ begin
41
+ require 'hodel_3000_compliant_logger'
42
+ config.logger = Hodel3000CompliantLogger.new(config.log_path)
43
+ rescue LoadError => e
44
+ $stderr.puts "Hodel3000CompliantLogger unavailable, oink will be disabled"
45
+ end
46
+ end
47
+
48
+
49
+ Oink allows you to add two type of statistics to your log output: memory usage and number of ActiveRecord objects instantiated. Here's an example for the output you can get:
50
+
51
+ Feb 08 11:39:54 ey33-s00302 rails[9076]: Memory usage: 316516 | PID: 9076
52
+ Feb 27 13:21:04 ey04-s00295 rails[11862]: Instantiation Breakdown: Total: 73 | User: 34 | Group: 20 | Medium: 20 | Sport: 10 | Post: 4 | Discussion: 2
53
+
54
+ For memory usage, add this to app/controllers/application_controller.rb:
55
+
56
+ class ApplicationController
57
+ include Oink::MemoryUsageLogger
58
+ end
59
+
60
+ For ActiveRecord objects instantiated, add this to your app/controllers/application_controller.rb:
61
+
62
+ class ApplicationController
63
+ include Oink::InstanceTypeCounter
64
+ end
65
+
66
+
67
+ == Analyizing logs
68
+
69
+ After installing the plugin and aggregating some enhanced logs, run the 'oink' executable against your server logs.
70
+
71
+ Usage: oink [options] files
72
+ -t, --threshold [INTEGER] Memory threshold in MB
73
+ -f, --file filepath Output to file
74
+ --format FORMAT Select format
75
+ (ss,v,s,verbose,short-summary,summary)
76
+ -m, --memory Check for Memory Threshold (default)
77
+ -r, --active-record Check for Active Record Threshold
78
+
79
+ Oink hunts for requests which exceed a given threshold. In "memory" mode (the default), the threshold represents a megabyte memory increase from the previous request. In "active record" mode (turned on by passing the --active-record switch), the threshold represents the number of active record objects instantiated during a request.
80
+
81
+ e.g. To find all actions which increase the heap size more than 75 MB, where log files are location in /tmp/logs/
82
+
83
+ $ oink --threshold=75 /tmp/logs/*
84
+ ---- MEMORY THRESHOLD ----
85
+ THRESHOLD: 75 MB
86
+
87
+ -- SUMMARY --
88
+ Worst Requests:
89
+ 1. Feb 02 16:26:06, 157524 KB, SportsController#show
90
+ 2. Feb 02 20:11:54, 134972 KB, DashboardsController#show
91
+ 3. Feb 02 19:06:13, 131912 KB, DashboardsController#show
92
+ 4. Feb 02 08:07:46, 115448 KB, GroupsController#show
93
+ 5. Feb 02 12:19:53, 112924 KB, GroupsController#show
94
+ 6. Feb 02 13:03:00, 112064 KB, ColorSchemesController#show
95
+ 7. Feb 02 13:01:59, 109148 KB, SessionsController#create
96
+ 8. Feb 02 06:11:17, 108456 KB, PublicPagesController#join
97
+ 9. Feb 02 08:43:06, 94468 KB, CommentsController#create
98
+ 10. Feb 02 20:49:44, 82340 KB, DashboardsController#show
99
+
100
+ Worst Actions:
101
+ 10, DashboardsController#show
102
+ 9, GroupsController#show
103
+ 5, PublicPagesController#show
104
+ 5, UsersController#show
105
+ 3, MediaController#show
106
+ 2, SportsController#show
107
+ 1, SessionsController#create
108
+ 1, GroupInvitesController#by_email
109
+ 1, MediaController#index
110
+ 1, PostsController#show
111
+ 1, PhotoVotesController#create
112
+ 1, AlbumsController#index
113
+ 1, SignupsController#new
114
+ 1, ColorSchemesController#show
115
+ 1, PublicPagesController#join
116
+ 1, CommentsController#create
117
+
118
+ e.g. In verbose mode, oink will print out all the log information from your logs about the actions which exceeded the threshold specified
119
+
120
+ $ oink --format verbose --threshold=75 /tmp/logs/*
121
+
122
+ ---------------------------------------------------------------------
123
+ Feb 08 11:39:52 ey33-s00302 rails[9076]: Processing UsersController#show (for 11.187.34.45 at 2009-02-08 11:39:52) [GET]
124
+ Feb 08 11:39:52 ey33-s00302 rails[9076]: Parameters: {"action"=>"show", "id"=>"45", "controller"=>"users"}
125
+ Feb 08 11:39:52 ey33-s00302 rails[9076]: Rendering template within layouts/application
126
+ Feb 08 11:39:52 ey33-s00302 rails[9076]: Rendering users/show
127
+ Feb 08 11:39:54 ey33-s00302 rails[9076]: Memory usage: 316516 | PID: 9076
128
+ Feb 08 11:39:54 ey33-s00302 rails[9076]: Completed in 2008ms (View: 1136, DB: 264) | 200 OK [http://www.example.com/users/45]
129
+ ---------------------------------------------------------------------
130
+
131
+ Verbose format prints the summary as well as each action which exceeded the threshold.
132
+
133
+ === Authors
134
+
135
+ - Maintained by Noah Davis
136
+ - Thanks to Weplay (http://weplay.com) for sponsoring development and supporting open sourcing it from the start
data/Rakefile ADDED
@@ -0,0 +1,29 @@
1
+ require 'rubygems'
2
+ require "rake/gempackagetask"
3
+ require "rake/clean"
4
+ require "spec/rake/spectask"
5
+ require './lib/oink/base.rb'
6
+
7
+ require 'jeweler'
8
+ Jeweler::Tasks.new do |s|
9
+ s.name = "oink"
10
+ s.version = Oink::Base::VERSION
11
+ s.author = "Noah Davis"
12
+ s.email = "noahd1" + "@" + "yahoo.com"
13
+ s.homepage = "http://github.com/noahd1/oink"
14
+ s.summary = "Log parser to identify actions which significantly increase VM heap size"
15
+ s.description = s.summary
16
+ s.executables = "oink"
17
+ s.files = %w[History.txt MIT-LICENSE README.rdoc Rakefile] + Dir["bin/*"] + Dir["lib/**/*"]
18
+ end
19
+
20
+ Jeweler::GemcutterTasks.new
21
+
22
+ Spec::Rake::SpecTask.new do |t|
23
+ t.spec_opts == ["--color"]
24
+ end
25
+
26
+ desc "Run the specs"
27
+ task :default => ["spec"]
28
+
29
+ CLEAN.include ["pkg", "*.gem", "doc", "ri", "coverage"]
data/bin/oink ADDED
@@ -0,0 +1,5 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ require File.dirname(__FILE__) + "/../lib/oink.rb"
4
+ Cli.new(ARGV.dup).process
5
+
data/lib/oink.rb ADDED
@@ -0,0 +1,9 @@
1
+ $:.unshift(File.dirname(__FILE__ + '.rb') + '/../lib') unless $:.include?(File.dirname(__FILE__ + '.rb') + '/../lib')
2
+
3
+ require "oink/memory_usage_reporter"
4
+ require "oink/active_record_instantiation_reporter"
5
+ require "oink/cli"
6
+
7
+ if defined?(Rails)
8
+ require 'oink/rails'
9
+ end
@@ -0,0 +1,68 @@
1
+ require "date"
2
+ require "oink/base"
3
+ require "oink/oinked_request/oinked_ar_request"
4
+ require "oink/priority_queue"
5
+
6
+ module Oink
7
+
8
+ class ActiveRecordInstantiationReporter < Base
9
+
10
+ def print(output)
11
+ output.puts "---- OINK FOR ACTIVERECORD ----"
12
+ output.puts "THRESHOLD: #{@threshold} Active Record objects per request\n"
13
+
14
+ output.puts "\n-- REQUESTS --\n" if @format == :verbose
15
+
16
+ @inputs.each do |input|
17
+ input.each_line do |line|
18
+ line = line.strip
19
+
20
+ # Skip this line since we're only interested in the Hodel 3000 compliant lines
21
+ next unless line =~ HODEL_LOG_FORMAT_REGEX
22
+
23
+ if line =~ /rails\[(\d+)\]/
24
+ pid = $1
25
+ @pids[pid] ||= { :buffer => [], :ar_count => -1, :action => "", :request_finished => true }
26
+ @pids[pid][:buffer] << line
27
+ end
28
+
29
+ if line =~ /Processing ((\w+)#(\w+)) /
30
+
31
+ @pids[pid][:action] = $1
32
+ unless @pids[pid][:request_finished]
33
+ @pids[pid][:buffer] = [line]
34
+ end
35
+ @pids[pid][:request_finished] = false
36
+
37
+ elsif line =~ /Instantiation Breakdown: Total: (\d+)/
38
+
39
+ @pids[pid][:ar_count] = $1.to_i
40
+
41
+ elsif line =~ /Completed in/
42
+
43
+ if @pids[pid][:ar_count] > @threshold
44
+ @bad_actions[@pids[pid][:action]] ||= 0
45
+ @bad_actions[@pids[pid][:action]] = @bad_actions[@pids[pid][:action]] + 1
46
+ date = HODEL_LOG_FORMAT_REGEX.match(line).captures[0]
47
+ @bad_requests.push(OinkedARRequest.new(@pids[pid][:action], date, @pids[pid][:buffer], @pids[pid][:ar_count]))
48
+ if @format == :verbose
49
+ @pids[pid][:buffer].each { |b| output.puts b }
50
+ output.puts "---------------------------------------------------------------------"
51
+ end
52
+ end
53
+
54
+ @pids[pid][:request_finished] = true
55
+ @pids[pid][:buffer] = []
56
+ @pids[pid][:ar_count] = -1
57
+
58
+ end # end elsif
59
+ end # end each_line
60
+ end # end each input
61
+
62
+ print_summary(output)
63
+
64
+ end
65
+
66
+ end
67
+
68
+ end
data/lib/oink/base.rb ADDED
@@ -0,0 +1,40 @@
1
+ module Oink
2
+
3
+ class Base
4
+
5
+ VERSION = '0.1.0'
6
+ FORMATS = %w[verbose short-summary summary]
7
+ FORMAT_ALIASES = { "v" => "verbose", "ss" => "short-summary", "s" => "summary" }
8
+ HODEL_LOG_FORMAT_REGEX = /^(\w+ \d{2} \d{2}:\d{2}:\d{2})/
9
+
10
+ def initialize(input, threshold, options = {})
11
+ @inputs = Array(input)
12
+ @threshold = threshold
13
+ @format = options[:format] || :short_summary
14
+
15
+ @pids = {}
16
+ @bad_actions = {}
17
+ @bad_requests = PriorityQueue.new(10)
18
+ end
19
+
20
+ protected
21
+
22
+ def print_summary(output)
23
+ output.puts "\n-- SUMMARY --\n"
24
+ output.puts "Worst Requests:"
25
+ @bad_requests.each_with_index do |offender, index|
26
+ output.puts "#{index + 1}. #{offender.datetime}, #{offender.display_oink_number}, #{offender.action}"
27
+ if @format == :summary
28
+ offender.log_lines.each { |b| output.puts b }
29
+ output.puts "---------------------------------------------------------------------"
30
+ end
31
+ end
32
+ output.puts "\nWorst Actions:"
33
+ @bad_actions.sort{|a,b| b[1]<=>a[1]}.each { |elem|
34
+ output.puts "#{elem[1]}, #{elem[0]}"
35
+ }
36
+ end
37
+
38
+ end
39
+
40
+ end
data/lib/oink/cli.rb ADDED
@@ -0,0 +1,97 @@
1
+ require 'optparse'
2
+
3
+ class Cli
4
+
5
+ def initialize(args)
6
+ @args = args
7
+ end
8
+
9
+ def process
10
+ options = { :format => :short_summary, :type => :memory }
11
+
12
+ op = OptionParser.new do |opts|
13
+ opts.banner = "Usage: oink [options] files"
14
+
15
+ opts.on("-t", "--threshold [INTEGER]", Integer,
16
+ "Memory threshold in MB") do |threshold|
17
+ options[:threshold] = threshold
18
+ end
19
+
20
+ opts.on("-f", "--file filepath", "Output to file") do |filename|
21
+ options[:output_file] = filename
22
+ end
23
+
24
+ format_list = (Oink::MemoryUsageReporter::FORMAT_ALIASES.keys + Oink::MemoryUsageReporter::FORMATS).join(',')
25
+ opts.on("--format FORMAT", Oink::MemoryUsageReporter::FORMATS, Oink::MemoryUsageReporter::FORMAT_ALIASES, "Select format",
26
+ " (#{format_list})") do |format|
27
+ options[:format] = format.to_sym
28
+ end
29
+
30
+ opts.on("-m", "--memory", "Check for Memory Threshold (default)") do |v|
31
+ options[:type] = :memory
32
+ end
33
+
34
+ opts.on("-r", "--active-record", "Check for Active Record Threshold") do |v|
35
+ options[:type] = :active_record
36
+ end
37
+
38
+ end
39
+
40
+ op.parse!(@args)
41
+
42
+ if @args.empty?
43
+ puts op
44
+ exit
45
+ end
46
+
47
+ output = nil
48
+
49
+ if options[:output_file]
50
+ output = File.open(options[:output_file], 'w')
51
+ else
52
+ output = STDOUT
53
+ end
54
+
55
+ files = get_file_listing(@args)
56
+
57
+ handles = files.map { |f| File.open(f) }
58
+
59
+ if options[:type] == :memory
60
+
61
+ options[:threshold] ||= 75
62
+ options[:threshold] *= 1024
63
+
64
+ Oink::MemoryUsageReporter.new(handles, options[:threshold], :format => options[:format]).print(output)
65
+
66
+ elsif options[:type] == :active_record
67
+
68
+ options[:threshold] ||= 500
69
+
70
+ Oink::ActiveRecordInstantiationReporter.new(handles, options[:threshold], :format => options[:format]).print(output)
71
+
72
+ end
73
+
74
+ output.close
75
+ handles.each { |h| h.close }
76
+ end
77
+
78
+ protected
79
+
80
+ def get_file_listing(args)
81
+ listing = []
82
+ args.each do |file|
83
+ unless File.exist?(file)
84
+ raise "Could not find \"#{file}\""
85
+ end
86
+ if File.directory?(file)
87
+ listing += Dir.glob("#{file}/**")
88
+ else
89
+ listing << file
90
+ end
91
+ end
92
+ listing
93
+ end
94
+
95
+ end
96
+
97
+
@@ -0,0 +1,72 @@
1
+ require "date"
2
+ require "oink/base"
3
+ require "oink/oinked_request/oinked_memory_request"
4
+ require "oink/priority_queue"
5
+
6
+ module Oink
7
+
8
+ class MemoryUsageReporter < Base
9
+ def print(output)
10
+ output.puts "---- MEMORY THRESHOLD ----"
11
+ output.puts "THRESHOLD: #{@threshold/1024} MB\n"
12
+
13
+ output.puts "\n-- REQUESTS --\n" if @format == :verbose
14
+
15
+ @inputs.each do |input|
16
+ input.each_line do |line|
17
+ line = line.strip
18
+
19
+ # Skip this line since we're only interested in the Hodel 3000 compliant lines
20
+ next unless line =~ HODEL_LOG_FORMAT_REGEX
21
+
22
+ if line =~ /rails\[(\d+)\]/
23
+ pid = $1
24
+ @pids[pid] ||= { :buffer => [], :last_memory_reading => -1, :current_memory_reading => -1, :action => "", :request_finished => true }
25
+ @pids[pid][:buffer] << line
26
+ end
27
+
28
+ if line =~ /Processing ((\w+)#(\w+)) /
29
+
30
+ unless @pids[pid][:request_finished]
31
+ @pids[pid][:last_memory_reading] = -1
32
+ end
33
+ @pids[pid][:action] = $1
34
+ @pids[pid][:request_finished] = false
35
+
36
+ elsif line =~ /Memory usage: (\d+) /
37
+
38
+ memory_reading = $1.to_i
39
+ @pids[pid][:current_memory_reading] = memory_reading
40
+
41
+ elsif line =~ /Completed in/
42
+
43
+ @pids[pid][:request_finished] = true
44
+ unless @pids[pid][:current_memory_reading] == -1 || @pids[pid][:last_memory_reading] == -1
45
+ memory_diff = @pids[pid][:current_memory_reading] - @pids[pid][:last_memory_reading]
46
+ if memory_diff > @threshold
47
+ @bad_actions[@pids[pid][:action]] ||= 0
48
+ @bad_actions[@pids[pid][:action]] = @bad_actions[@pids[pid][:action]] + 1
49
+ date = HODEL_LOG_FORMAT_REGEX.match(line).captures[0]
50
+ @bad_requests.push(OinkedMemoryRequest.new(@pids[pid][:action], date, @pids[pid][:buffer], memory_diff))
51
+ if @format == :verbose
52
+ @pids[pid][:buffer].each { |b| output.puts b }
53
+ output.puts "---------------------------------------------------------------------"
54
+ end
55
+ end
56
+ end
57
+
58
+ @pids[pid][:buffer] = []
59
+ @pids[pid][:last_memory_reading] = @pids[pid][:current_memory_reading]
60
+ @pids[pid][:current_memory_reading] = -1
61
+
62
+ end # end elsif
63
+ end # end each_line
64
+ end # end each input
65
+
66
+ print_summary(output)
67
+
68
+ end
69
+
70
+ end
71
+
72
+ end