oink 0.1.0

Sign up to get free protection for your applications and to get access to all the features.
data/History.txt ADDED
@@ -0,0 +1,4 @@
1
+ === 0.1.0 / 2009-02-09
2
+
3
+ * 1st release
4
+
data/MIT-LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ Copyright (c) 2009 Noah Davis
2
+ Copyright (c) 2008 Ben Johnson of Binary Logic (binarylogic.com)
3
+
4
+ Permission is hereby granted, free of charge, to any person obtaining
5
+ a copy of this software and associated documentation files (the
6
+ "Software"), to deal in the Software without restriction, including
7
+ without limitation the rights to use, copy, modify, merge, publish,
8
+ distribute, sublicense, and/or sell copies of the Software, and to
9
+ permit persons to whom the Software is furnished to do so, subject to
10
+ the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be
13
+ included in all copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
16
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
17
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
18
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
19
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
20
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
21
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
data/README.rdoc ADDED
@@ -0,0 +1,136 @@
1
+ === Oink
2
+
3
+ * http://github.com/noahd1/oink
4
+
5
+ === Description
6
+
7
+ Rails plugin and log parser to help narrow down the source(s) of increased memory usage in rails applications.
8
+
9
+ === Synopsis
10
+
11
+ Oink adds memory and active record instantiation information to rails log during runtime and provides an executable to help digest the enhanced logs.
12
+
13
+ Given a minimum threshold and a metric (memory or active record instantiation), the oink executable reports:
14
+
15
+ 1. The top ten single requests which exceeded the threshold for the metric, ordered by the request which exceeded the threshold the most
16
+ 2. The number of times each action exceeded the threshold for the metric, ordered by the action which exceeded the threshold the most
17
+ 3. (in verbose mode) The log lines produced by requests which exceeded the threshold
18
+
19
+ Many thanks to Ben Johnson for Memory Usage Logger (http://github.com/binarylogic/memory_usage_logger/tree/master) which is the basis of this plugin.
20
+
21
+ == Dependencies
22
+
23
+ Currently oink can only parse logs in the Hodel3000Logger format
24
+
25
+ - http://github.com/topfunky/hodel_3000_compliant_logger
26
+
27
+ When used as a gem, this is automatically brought in as a dependency.
28
+
29
+ === Installation and configuration
30
+
31
+ To begin, you'll want to install it:
32
+
33
+ gem install oink
34
+
35
+ Next, you'll need to update your config/environment.rb to include oink as a dependency, and to configure Rails to use the hodel_3000_compliant_logger:
36
+
37
+ Rails::Initializer.run do |config|
38
+ # your other config here
39
+ config.gem 'oink'
40
+ begin
41
+ require 'hodel_3000_compliant_logger'
42
+ config.logger = Hodel3000CompliantLogger.new(config.log_path)
43
+ rescue LoadError => e
44
+ $stderr.puts "Hodel3000CompliantLogger unavailable, oink will be disabled"
45
+ end
46
+ end
47
+
48
+
49
+ Oink allows you to add two type of statistics to your log output: memory usage and number of ActiveRecord objects instantiated. Here's an example for the output you can get:
50
+
51
+ Feb 08 11:39:54 ey33-s00302 rails[9076]: Memory usage: 316516 | PID: 9076
52
+ Feb 27 13:21:04 ey04-s00295 rails[11862]: Instantiation Breakdown: Total: 73 | User: 34 | Group: 20 | Medium: 20 | Sport: 10 | Post: 4 | Discussion: 2
53
+
54
+ For memory usage, add this to app/controllers/application_controller.rb:
55
+
56
+ class ApplicationController
57
+ include Oink::MemoryUsageLogger
58
+ end
59
+
60
+ For ActiveRecord objects instantiated, add this to your app/controllers/application_controller.rb:
61
+
62
+ class ApplicationController
63
+ include Oink::InstanceTypeCounter
64
+ end
65
+
66
+
67
+ == Analyizing logs
68
+
69
+ After installing the plugin and aggregating some enhanced logs, run the 'oink' executable against your server logs.
70
+
71
+ Usage: oink [options] files
72
+ -t, --threshold [INTEGER] Memory threshold in MB
73
+ -f, --file filepath Output to file
74
+ --format FORMAT Select format
75
+ (ss,v,s,verbose,short-summary,summary)
76
+ -m, --memory Check for Memory Threshold (default)
77
+ -r, --active-record Check for Active Record Threshold
78
+
79
+ Oink hunts for requests which exceed a given threshold. In "memory" mode (the default), the threshold represents a megabyte memory increase from the previous request. In "active record" mode (turned on by passing the --active-record switch), the threshold represents the number of active record objects instantiated during a request.
80
+
81
+ e.g. To find all actions which increase the heap size more than 75 MB, where log files are location in /tmp/logs/
82
+
83
+ $ oink --threshold=75 /tmp/logs/*
84
+ ---- MEMORY THRESHOLD ----
85
+ THRESHOLD: 75 MB
86
+
87
+ -- SUMMARY --
88
+ Worst Requests:
89
+ 1. Feb 02 16:26:06, 157524 KB, SportsController#show
90
+ 2. Feb 02 20:11:54, 134972 KB, DashboardsController#show
91
+ 3. Feb 02 19:06:13, 131912 KB, DashboardsController#show
92
+ 4. Feb 02 08:07:46, 115448 KB, GroupsController#show
93
+ 5. Feb 02 12:19:53, 112924 KB, GroupsController#show
94
+ 6. Feb 02 13:03:00, 112064 KB, ColorSchemesController#show
95
+ 7. Feb 02 13:01:59, 109148 KB, SessionsController#create
96
+ 8. Feb 02 06:11:17, 108456 KB, PublicPagesController#join
97
+ 9. Feb 02 08:43:06, 94468 KB, CommentsController#create
98
+ 10. Feb 02 20:49:44, 82340 KB, DashboardsController#show
99
+
100
+ Worst Actions:
101
+ 10, DashboardsController#show
102
+ 9, GroupsController#show
103
+ 5, PublicPagesController#show
104
+ 5, UsersController#show
105
+ 3, MediaController#show
106
+ 2, SportsController#show
107
+ 1, SessionsController#create
108
+ 1, GroupInvitesController#by_email
109
+ 1, MediaController#index
110
+ 1, PostsController#show
111
+ 1, PhotoVotesController#create
112
+ 1, AlbumsController#index
113
+ 1, SignupsController#new
114
+ 1, ColorSchemesController#show
115
+ 1, PublicPagesController#join
116
+ 1, CommentsController#create
117
+
118
+ e.g. In verbose mode, oink will print out all the log information from your logs about the actions which exceeded the threshold specified
119
+
120
+ $ oink --format verbose --threshold=75 /tmp/logs/*
121
+
122
+ ---------------------------------------------------------------------
123
+ Feb 08 11:39:52 ey33-s00302 rails[9076]: Processing UsersController#show (for 11.187.34.45 at 2009-02-08 11:39:52) [GET]
124
+ Feb 08 11:39:52 ey33-s00302 rails[9076]: Parameters: {"action"=>"show", "id"=>"45", "controller"=>"users"}
125
+ Feb 08 11:39:52 ey33-s00302 rails[9076]: Rendering template within layouts/application
126
+ Feb 08 11:39:52 ey33-s00302 rails[9076]: Rendering users/show
127
+ Feb 08 11:39:54 ey33-s00302 rails[9076]: Memory usage: 316516 | PID: 9076
128
+ Feb 08 11:39:54 ey33-s00302 rails[9076]: Completed in 2008ms (View: 1136, DB: 264) | 200 OK [http://www.example.com/users/45]
129
+ ---------------------------------------------------------------------
130
+
131
+ Verbose format prints the summary as well as each action which exceeded the threshold.
132
+
133
+ === Authors
134
+
135
+ - Maintained by Noah Davis
136
+ - Thanks to Weplay (http://weplay.com) for sponsoring development and supporting open sourcing it from the start
data/Rakefile ADDED
@@ -0,0 +1,29 @@
1
+ require 'rubygems'
2
+ require "rake/gempackagetask"
3
+ require "rake/clean"
4
+ require "spec/rake/spectask"
5
+ require './lib/oink/base.rb'
6
+
7
+ require 'jeweler'
8
+ Jeweler::Tasks.new do |s|
9
+ s.name = "oink"
10
+ s.version = Oink::Base::VERSION
11
+ s.author = "Noah Davis"
12
+ s.email = "noahd1" + "@" + "yahoo.com"
13
+ s.homepage = "http://github.com/noahd1/oink"
14
+ s.summary = "Log parser to identify actions which significantly increase VM heap size"
15
+ s.description = s.summary
16
+ s.executables = "oink"
17
+ s.files = %w[History.txt MIT-LICENSE README.rdoc Rakefile] + Dir["bin/*"] + Dir["lib/**/*"]
18
+ end
19
+
20
+ Jeweler::GemcutterTasks.new
21
+
22
+ Spec::Rake::SpecTask.new do |t|
23
+ t.spec_opts == ["--color"]
24
+ end
25
+
26
+ desc "Run the specs"
27
+ task :default => ["spec"]
28
+
29
+ CLEAN.include ["pkg", "*.gem", "doc", "ri", "coverage"]
data/bin/oink ADDED
@@ -0,0 +1,5 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ require File.dirname(__FILE__) + "/../lib/oink.rb"
4
+ Cli.new(ARGV.dup).process
5
+
data/lib/oink.rb ADDED
@@ -0,0 +1,9 @@
1
+ $:.unshift(File.dirname(__FILE__ + '.rb') + '/../lib') unless $:.include?(File.dirname(__FILE__ + '.rb') + '/../lib')
2
+
3
+ require "oink/memory_usage_reporter"
4
+ require "oink/active_record_instantiation_reporter"
5
+ require "oink/cli"
6
+
7
+ if defined?(Rails)
8
+ require 'oink/rails'
9
+ end
@@ -0,0 +1,68 @@
1
+ require "date"
2
+ require "oink/base"
3
+ require "oink/oinked_request/oinked_ar_request"
4
+ require "oink/priority_queue"
5
+
6
+ module Oink
7
+
8
+ class ActiveRecordInstantiationReporter < Base
9
+
10
+ def print(output)
11
+ output.puts "---- OINK FOR ACTIVERECORD ----"
12
+ output.puts "THRESHOLD: #{@threshold} Active Record objects per request\n"
13
+
14
+ output.puts "\n-- REQUESTS --\n" if @format == :verbose
15
+
16
+ @inputs.each do |input|
17
+ input.each_line do |line|
18
+ line = line.strip
19
+
20
+ # Skip this line since we're only interested in the Hodel 3000 compliant lines
21
+ next unless line =~ HODEL_LOG_FORMAT_REGEX
22
+
23
+ if line =~ /rails\[(\d+)\]/
24
+ pid = $1
25
+ @pids[pid] ||= { :buffer => [], :ar_count => -1, :action => "", :request_finished => true }
26
+ @pids[pid][:buffer] << line
27
+ end
28
+
29
+ if line =~ /Processing ((\w+)#(\w+)) /
30
+
31
+ @pids[pid][:action] = $1
32
+ unless @pids[pid][:request_finished]
33
+ @pids[pid][:buffer] = [line]
34
+ end
35
+ @pids[pid][:request_finished] = false
36
+
37
+ elsif line =~ /Instantiation Breakdown: Total: (\d+)/
38
+
39
+ @pids[pid][:ar_count] = $1.to_i
40
+
41
+ elsif line =~ /Completed in/
42
+
43
+ if @pids[pid][:ar_count] > @threshold
44
+ @bad_actions[@pids[pid][:action]] ||= 0
45
+ @bad_actions[@pids[pid][:action]] = @bad_actions[@pids[pid][:action]] + 1
46
+ date = HODEL_LOG_FORMAT_REGEX.match(line).captures[0]
47
+ @bad_requests.push(OinkedARRequest.new(@pids[pid][:action], date, @pids[pid][:buffer], @pids[pid][:ar_count]))
48
+ if @format == :verbose
49
+ @pids[pid][:buffer].each { |b| output.puts b }
50
+ output.puts "---------------------------------------------------------------------"
51
+ end
52
+ end
53
+
54
+ @pids[pid][:request_finished] = true
55
+ @pids[pid][:buffer] = []
56
+ @pids[pid][:ar_count] = -1
57
+
58
+ end # end elsif
59
+ end # end each_line
60
+ end # end each input
61
+
62
+ print_summary(output)
63
+
64
+ end
65
+
66
+ end
67
+
68
+ end
data/lib/oink/base.rb ADDED
@@ -0,0 +1,40 @@
1
+ module Oink
2
+
3
+ class Base
4
+
5
+ VERSION = '0.1.0'
6
+ FORMATS = %w[verbose short-summary summary]
7
+ FORMAT_ALIASES = { "v" => "verbose", "ss" => "short-summary", "s" => "summary" }
8
+ HODEL_LOG_FORMAT_REGEX = /^(\w+ \d{2} \d{2}:\d{2}:\d{2})/
9
+
10
+ def initialize(input, threshold, options = {})
11
+ @inputs = Array(input)
12
+ @threshold = threshold
13
+ @format = options[:format] || :short_summary
14
+
15
+ @pids = {}
16
+ @bad_actions = {}
17
+ @bad_requests = PriorityQueue.new(10)
18
+ end
19
+
20
+ protected
21
+
22
+ def print_summary(output)
23
+ output.puts "\n-- SUMMARY --\n"
24
+ output.puts "Worst Requests:"
25
+ @bad_requests.each_with_index do |offender, index|
26
+ output.puts "#{index + 1}. #{offender.datetime}, #{offender.display_oink_number}, #{offender.action}"
27
+ if @format == :summary
28
+ offender.log_lines.each { |b| output.puts b }
29
+ output.puts "---------------------------------------------------------------------"
30
+ end
31
+ end
32
+ output.puts "\nWorst Actions:"
33
+ @bad_actions.sort{|a,b| b[1]<=>a[1]}.each { |elem|
34
+ output.puts "#{elem[1]}, #{elem[0]}"
35
+ }
36
+ end
37
+
38
+ end
39
+
40
+ end
data/lib/oink/cli.rb ADDED
@@ -0,0 +1,97 @@
1
+ require 'optparse'
2
+
3
+ class Cli
4
+
5
+ def initialize(args)
6
+ @args = args
7
+ end
8
+
9
+ def process
10
+ options = { :format => :short_summary, :type => :memory }
11
+
12
+ op = OptionParser.new do |opts|
13
+ opts.banner = "Usage: oink [options] files"
14
+
15
+ opts.on("-t", "--threshold [INTEGER]", Integer,
16
+ "Memory threshold in MB") do |threshold|
17
+ options[:threshold] = threshold
18
+ end
19
+
20
+ opts.on("-f", "--file filepath", "Output to file") do |filename|
21
+ options[:output_file] = filename
22
+ end
23
+
24
+ format_list = (Oink::MemoryUsageReporter::FORMAT_ALIASES.keys + Oink::MemoryUsageReporter::FORMATS).join(',')
25
+ opts.on("--format FORMAT", Oink::MemoryUsageReporter::FORMATS, Oink::MemoryUsageReporter::FORMAT_ALIASES, "Select format",
26
+ " (#{format_list})") do |format|
27
+ options[:format] = format.to_sym
28
+ end
29
+
30
+ opts.on("-m", "--memory", "Check for Memory Threshold (default)") do |v|
31
+ options[:type] = :memory
32
+ end
33
+
34
+ opts.on("-r", "--active-record", "Check for Active Record Threshold") do |v|
35
+ options[:type] = :active_record
36
+ end
37
+
38
+ end
39
+
40
+ op.parse!(@args)
41
+
42
+ if @args.empty?
43
+ puts op
44
+ exit
45
+ end
46
+
47
+ output = nil
48
+
49
+ if options[:output_file]
50
+ output = File.open(options[:output_file], 'w')
51
+ else
52
+ output = STDOUT
53
+ end
54
+
55
+ files = get_file_listing(@args)
56
+
57
+ handles = files.map { |f| File.open(f) }
58
+
59
+ if options[:type] == :memory
60
+
61
+ options[:threshold] ||= 75
62
+ options[:threshold] *= 1024
63
+
64
+ Oink::MemoryUsageReporter.new(handles, options[:threshold], :format => options[:format]).print(output)
65
+
66
+ elsif options[:type] == :active_record
67
+
68
+ options[:threshold] ||= 500
69
+
70
+ Oink::ActiveRecordInstantiationReporter.new(handles, options[:threshold], :format => options[:format]).print(output)
71
+
72
+ end
73
+
74
+ output.close
75
+ handles.each { |h| h.close }
76
+ end
77
+
78
+ protected
79
+
80
+ def get_file_listing(args)
81
+ listing = []
82
+ args.each do |file|
83
+ unless File.exist?(file)
84
+ raise "Could not find \"#{file}\""
85
+ end
86
+ if File.directory?(file)
87
+ listing += Dir.glob("#{file}/**")
88
+ else
89
+ listing << file
90
+ end
91
+ end
92
+ listing
93
+ end
94
+
95
+ end
96
+
97
+
@@ -0,0 +1,72 @@
1
+ require "date"
2
+ require "oink/base"
3
+ require "oink/oinked_request/oinked_memory_request"
4
+ require "oink/priority_queue"
5
+
6
+ module Oink
7
+
8
+ class MemoryUsageReporter < Base
9
+ def print(output)
10
+ output.puts "---- MEMORY THRESHOLD ----"
11
+ output.puts "THRESHOLD: #{@threshold/1024} MB\n"
12
+
13
+ output.puts "\n-- REQUESTS --\n" if @format == :verbose
14
+
15
+ @inputs.each do |input|
16
+ input.each_line do |line|
17
+ line = line.strip
18
+
19
+ # Skip this line since we're only interested in the Hodel 3000 compliant lines
20
+ next unless line =~ HODEL_LOG_FORMAT_REGEX
21
+
22
+ if line =~ /rails\[(\d+)\]/
23
+ pid = $1
24
+ @pids[pid] ||= { :buffer => [], :last_memory_reading => -1, :current_memory_reading => -1, :action => "", :request_finished => true }
25
+ @pids[pid][:buffer] << line
26
+ end
27
+
28
+ if line =~ /Processing ((\w+)#(\w+)) /
29
+
30
+ unless @pids[pid][:request_finished]
31
+ @pids[pid][:last_memory_reading] = -1
32
+ end
33
+ @pids[pid][:action] = $1
34
+ @pids[pid][:request_finished] = false
35
+
36
+ elsif line =~ /Memory usage: (\d+) /
37
+
38
+ memory_reading = $1.to_i
39
+ @pids[pid][:current_memory_reading] = memory_reading
40
+
41
+ elsif line =~ /Completed in/
42
+
43
+ @pids[pid][:request_finished] = true
44
+ unless @pids[pid][:current_memory_reading] == -1 || @pids[pid][:last_memory_reading] == -1
45
+ memory_diff = @pids[pid][:current_memory_reading] - @pids[pid][:last_memory_reading]
46
+ if memory_diff > @threshold
47
+ @bad_actions[@pids[pid][:action]] ||= 0
48
+ @bad_actions[@pids[pid][:action]] = @bad_actions[@pids[pid][:action]] + 1
49
+ date = HODEL_LOG_FORMAT_REGEX.match(line).captures[0]
50
+ @bad_requests.push(OinkedMemoryRequest.new(@pids[pid][:action], date, @pids[pid][:buffer], memory_diff))
51
+ if @format == :verbose
52
+ @pids[pid][:buffer].each { |b| output.puts b }
53
+ output.puts "---------------------------------------------------------------------"
54
+ end
55
+ end
56
+ end
57
+
58
+ @pids[pid][:buffer] = []
59
+ @pids[pid][:last_memory_reading] = @pids[pid][:current_memory_reading]
60
+ @pids[pid][:current_memory_reading] = -1
61
+
62
+ end # end elsif
63
+ end # end each_line
64
+ end # end each input
65
+
66
+ print_summary(output)
67
+
68
+ end
69
+
70
+ end
71
+
72
+ end