aiwilliams-oink 0.1.0

data/History.txt

@@ -0,0 +1,4 @@
+=== 0.1.0 / 2009-02-09
+
+* 1st release
+

data/MIT-LICENSE

@@ -0,0 +1,21 @@
+Copyright (c) 2009 Noah Davis
+Copyright (c) 2008 Ben Johnson of Binary Logic (binarylogic.com)
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.rdoc

@@ -0,0 +1,118 @@
+=== Oink
+
+* http://github.com/noahd1/oink
+
+=== Description
+
+Rails plugin and log parser to help narrow down the source(s) of increased memory usage in rails applications.
+
+=== Synopsis
+
+Oink adds memory and active record instantiation information to rails log during runtime and provides an executable to help digest the enhanced logs.
+
+Given a minimum threshold and a metric (memory or active record instantiation), the oink executable reports:
+
+1. The top ten single requests which exceeded the threshold for the metric, ordered by the request which exceeded the threshold the most
+2. The number of times each action exceeded the threshold for the metric, ordered by the action which exceeded the threshold the most
+3. (in verbose mode) The log lines produced by requests which exceeded the threshold
+
+Many thanks to Ben Johnson for Memory Usage Logger (http://github.com/binarylogic/memory_usage_logger/tree/master) which is the basis of this plugin.
+
+== Dependencies
+
+Currently oink can only parse logs in the Hodel3000Logger format
+
+- http://github.com/topfunky/hodel_3000_compliant_logger/tree/master
+
+=== Install
+
+To print memory usage at the end of each request, include Oink::MemoryUsageLogger in your controller. This will add lines like:
+
+class ApplicationController
+  include Oink::MemoryUsageLogger
+end
+
+which results in lines like this added to each request:
+
+  Feb 08 11:39:54 ey33-s00302 rails[9076]: Memory usage: 316516 | PID: 9076
+
+To print active record instantiation at the end of each request, include Oink::InstanceTypeCounter in your controller. 
+
+  class ApplicationController
+    include Oink::InstanceTypeCounter
+  end
+
+which results in lines like this added to each request:
+
+  Feb 27 13:21:04 ey04-s00295 rails[11862]: Instantiation Breakdown: Total: 73 | User: 34 | Group: 20 | Medium: 20 | Sport: 10 | Post: 4 | Discussion: 2
+
+== Usage
+
+After installing the plugin and aggregating some enhanced logs, run the oink executable against the logs. Oink can point at a log file or a directory of log files
+
+  Usage: oink [options] files
+      -t, --threshold [INTEGER]        Memory threshold in MB
+      -f, --file filepath              Output to file
+          --format FORMAT              Select format
+                                         (ss,v,s,verbose,short-summary,summary)
+      -m, --memory                     Check for Memory Threshold (default)
+      -r, --active-record              Check for Active Record Threshold
+
+Oink hunts for requests which exceed a given threshold. In "memory" mode (the default), the threshold represents a megabyte memory increase from the previous request. In "active record" mode (turned on by passing the --active-record switch), the threshold represents the number of active record objects instantiated during a request.
+
+e.g. To find all actions which increase the heap size more than 75 MB, where log files are location in /tmp/logs/
+
+  $ oink --threshold=75 /tmp/logs/*
+  ---- MEMORY THRESHOLD ----
+  THRESHOLD: 75 MB
+
+  -- SUMMARY --
+  Worst Requests:
+  1. Feb 02 16:26:06, 157524 KB, SportsController#show
+  2. Feb 02 20:11:54, 134972 KB, DashboardsController#show
+  3. Feb 02 19:06:13, 131912 KB, DashboardsController#show
+  4. Feb 02 08:07:46, 115448 KB, GroupsController#show
+  5. Feb 02 12:19:53, 112924 KB, GroupsController#show
+  6. Feb 02 13:03:00, 112064 KB, ColorSchemesController#show
+  7. Feb 02 13:01:59, 109148 KB, SessionsController#create
+  8. Feb 02 06:11:17, 108456 KB, PublicPagesController#join
+  9. Feb 02 08:43:06, 94468 KB, CommentsController#create
+  10. Feb 02 20:49:44, 82340 KB, DashboardsController#show
+
+  Worst Actions:
+  10, DashboardsController#show
+  9, GroupsController#show
+  5, PublicPagesController#show
+  5, UsersController#show
+  3, MediaController#show
+  2, SportsController#show
+  1, SessionsController#create
+  1, GroupInvitesController#by_email
+  1, MediaController#index
+  1, PostsController#show
+  1, PhotoVotesController#create
+  1, AlbumsController#index
+  1, SignupsController#new
+  1, ColorSchemesController#show
+  1, PublicPagesController#join
+  1, CommentsController#create
+
+e.g. In verbose mode, oink will print out all the log information from your logs about the actions which exceeded the threshold specified
+
+  $ oink --verbose --threshold=75 /tmp/logs/*
+
+  ---------------------------------------------------------------------
+  Feb 08 11:39:52 ey33-s00302 rails[9076]: Processing UsersController#show (for 11.187.34.45 at 2009-02-08 11:39:52) [GET]
+  Feb 08 11:39:52 ey33-s00302 rails[9076]: Parameters: {"action"=>"show", "id"=>"45", "controller"=>"users"}
+  Feb 08 11:39:52 ey33-s00302 rails[9076]: Rendering template within layouts/application
+  Feb 08 11:39:52 ey33-s00302 rails[9076]: Rendering users/show
+  Feb 08 11:39:54 ey33-s00302 rails[9076]: Memory usage: 316516 | PID: 9076
+  Feb 08 11:39:54 ey33-s00302 rails[9076]: Completed in 2008ms (View: 1136, DB: 264) | 200 OK [http://www.example.com/users/45]
+  ---------------------------------------------------------------------
+
+Verbose format prints the summary as well as each action which exceeded the threshold.
+
+=== Authors
+
+- Maintained by Noah Davis
+- Thanks to Weplay (http://weplay.com) for sponsoring development and supporting open sourcing it from the start

data/Rakefile

@@ -0,0 +1,44 @@
+require 'rubygems'
+require "rake/gempackagetask"
+require "rake/clean"
+require "spec/rake/spectask"
+require './lib/oink/base.rb'
+
+spec = Gem::Specification.new do |s|
+  s.name         = "oink"
+  s.version      = Oink::Base::VERSION
+  s.author       = "Noah Davis"
+  s.email        = "noahd1" + "@" + "yahoo.com"
+  s.homepage     = "http://github.com/noahd1/oink"
+  s.summary      = "Log parser to identify actions which significantly increase VM heap size"
+  s.description  = s.summary
+  s.executables  = "oink"
+  s.files        = %w[History.txt MIT-LICENSE README.rdoc Rakefile] + Dir["bin/*"] + Dir["lib/**/*"]
+end
+
+Spec::Rake::SpecTask.new do |t|
+  t.spec_opts == ["--color"]
+end
+
+Rake::GemPackageTask.new(spec) do |package|
+  package.gem_spec = spec
+end
+
+desc "Run the specs"
+task :default => ["spec"]
+
+desc 'Show information about the gem.'
+task :write_gemspec do
+  File.open("oink.gemspec", 'w') do |f|
+    f.write spec.to_ruby
+  end
+  puts "Generated: oink.gemspec"
+end
+
+CLEAN.include ["pkg", "*.gem", "doc", "ri", "coverage"]
+
+desc 'Install the package as a gem.'
+task :install_gem => [:clean, :package] do
+  gem = Dir['pkg/*.gem'].first
+  sh "sudo gem install --local #{gem}"
+end

data/bin/oink

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+
+require File.dirname(__FILE__) + "/../lib/oink.rb"
+Cli.new(ARGV.dup).process
+

data/lib/oink.rb

@@ -0,0 +1,5 @@
+$:.unshift(File.dirname(__FILE__ + '.rb') + '/../lib') unless $:.include?(File.dirname(__FILE__ + '.rb') + '/../lib')
+
+require "oink/memory_usage_reporter"
+require "oink/active_record_instantiation_reporter"
+require "oink/cli"

data/lib/oink/active_record_instantiation_reporter.rb

@@ -0,0 +1,65 @@
+require "date"
+require "oink/base"
+require "oink/oinked_request/oinked_ar_request"
+require "oink/priority_queue"
+
+module Oink
+
+  class ActiveRecordInstantiationReporter < Base
+  
+    def print(output)
+      output.puts "---- OINK FOR ACTIVERECORD ----"
+      output.puts "THRESHOLD: #{@threshold} Active Record objects per request\n"
+    
+      output.puts "\n-- REQUESTS --\n" if @format == :verbose
+    
+      @inputs.each do |input|
+        input.each_line do |line|
+          line = line.strip
+
+          if line =~ /rails\[(\d+)\]/
+            pid = $1
+            @pids[pid] ||= { :buffer => [], :ar_count => -1, :action => "", :request_finished => true }
+            @pids[pid][:buffer] << line
+          end
+
+          if line =~ /Processing ((\w+)#(\w+)) /
+
+            @pids[pid][:action] = $1
+            unless @pids[pid][:request_finished]
+              @pids[pid][:buffer] = [line]
+            end
+            @pids[pid][:request_finished] = false
+      
+          elsif line =~ /Instantiation Breakdown: Total: (\d+)/
+
+            @pids[pid][:ar_count] = $1.to_i
+      
+          elsif line =~ /Completed in/
+
+            if @pids[pid][:ar_count] > @threshold
+              @bad_actions[@pids[pid][:action]] ||= 0
+              @bad_actions[@pids[pid][:action]] = @bad_actions[@pids[pid][:action]] + 1
+              date = /^(\w+ \d{2} \d{2}:\d{2}:\d{2})/.match(line).captures[0]
+              @bad_requests.push(OinkedARRequest.new(@pids[pid][:action], date, @pids[pid][:buffer], @pids[pid][:ar_count]))
+              if @format == :verbose
+                @pids[pid][:buffer].each { |b| output.puts b } 
+                output.puts "---------------------------------------------------------------------"
+              end
+            end
+          
+            @pids[pid][:request_finished] = true
+            @pids[pid][:buffer] = []
+            @pids[pid][:ar_count] = -1
+        
+          end # end elsif
+        end # end each_line
+      end # end each input
+
+      print_summary(output)
+    
+    end
+
+  end
+  
+end

data/lib/oink/base.rb

@@ -0,0 +1,39 @@
+module Oink
+
+  class Base
+
+    VERSION = '0.1.0'
+    FORMATS = %w[verbose short-summary summary]
+    FORMAT_ALIASES = { "v" => "verbose", "ss" => "short-summary", "s" => "summary" }
+
+    def initialize(input, threshold, options = {})
+      @inputs = Array(input)
+      @threshold = threshold
+      @format = options[:format] || :short_summary
+  
+      @pids = {}
+      @bad_actions = {}
+      @bad_requests = PriorityQueue.new(10)
+    end
+
+  protected
+
+    def print_summary(output)
+      output.puts "\n-- SUMMARY --\n"
+      output.puts "Worst Requests:"
+      @bad_requests.each_with_index do |offender, index|
+        output.puts "#{index + 1}. #{offender.datetime}, #{offender.display_oink_number}, #{offender.action}"
+        if @format == :summary
+          offender.log_lines.each { |b| output.puts b } 
+          output.puts "---------------------------------------------------------------------"
+        end
+      end
+      output.puts "\nWorst Actions:"
+      @bad_actions.sort{|a,b| b[1]<=>a[1]}.each { |elem|
+        output.puts "#{elem[1]}, #{elem[0]}"
+      }
+    end
+
+  end
+  
+end

data/lib/oink/cli.rb

@@ -0,0 +1,97 @@
+require 'optparse'
+
+class Cli
+  
+  def initialize(args)
+    @args = args
+  end
+  
+  def process
+    options = { :format => :short_summary, :type => :memory }
+
+    op = OptionParser.new do |opts|
+      opts.banner = "Usage: oink [options] files"
+
+      opts.on("-t", "--threshold [INTEGER]", Integer,
+              "Memory threshold in MB") do |threshold|
+        options[:threshold] = threshold
+      end
+
+      opts.on("-f", "--file filepath", "Output to file") do |filename|
+        options[:output_file] = filename
+      end
+
+      format_list = (Oink::MemoryUsageReporter::FORMAT_ALIASES.keys + Oink::MemoryUsageReporter::FORMATS).join(',')
+      opts.on("--format FORMAT", Oink::MemoryUsageReporter::FORMATS, Oink::MemoryUsageReporter::FORMAT_ALIASES, "Select format",
+              "  (#{format_list})") do |format|
+        options[:format] = format.to_sym
+      end
+
+      opts.on("-m", "--memory", "Check for Memory Threshold (default)") do |v|
+         options[:type] = :memory
+      end
+
+      opts.on("-r", "--active-record", "Check for Active Record Threshold") do |v|
+         options[:type] = :active_record
+      end
+
+    end
+
+    op.parse!(@args)
+
+    if @args.empty?
+      puts op
+      exit
+    end
+
+    output = nil
+
+    if options[:output_file]
+      output = File.open(options[:output_file], 'w')
+    else
+      output = STDOUT
+    end
+
+    files = get_file_listing(@args)
+
+    handles = files.map { |f| File.open(f) }
+
+    if options[:type] == :memory
+
+      options[:threshold] ||= 75
+      options[:threshold] *= 1024
+
+      Oink::MemoryUsageReporter.new(handles, options[:threshold], :format => options[:format]).print(output)
+
+    elsif options[:type] == :active_record
+
+      options[:threshold] ||= 500
+
+      Oink::ActiveRecordInstantiationReporter.new(handles, options[:threshold], :format => options[:format]).print(output)
+
+    end
+
+    output.close
+    handles.each { |h| h.close }    
+  end
+  
+protected
+
+  def get_file_listing(args)
+    listing = []
+    args.each do |file|
+      unless File.exist?(file)
+        raise "Could not find \"#{file}\""
+      end
+      if File.directory?(file)
+        listing += Dir.glob("#{file}/**")
+      else
+        listing << file
+      end
+    end
+    listing
+  end
+
+end
+
+

data/lib/oink/memory_usage_reporter.rb

@@ -0,0 +1,70 @@
+require "date"
+require "oink/base"
+require "oink/oinked_request/oinked_memory_request"
+require "oink/priority_queue"
+
+module Oink
+
+  class MemoryUsageReporter < Base
+
+    def print(output)
+      output.puts "---- MEMORY THRESHOLD ----"
+      output.puts "THRESHOLD: #{@threshold/1024} MB\n"
+    
+      output.puts "\n-- REQUESTS --\n" if @format == :verbose
+    
+      @inputs.each do |input|
+        input.each_line do |line|
+          line = line.strip
+
+          if line =~ /rails\[(\d+)\]/
+            pid = $1
+            @pids[pid] ||= { :buffer => [], :last_memory_reading => -1, :current_memory_reading => -1, :action => "", :request_finished => true }
+            @pids[pid][:buffer] << line
+          end
+
+          if line =~ /Processing ((\w+)#(\w+)) /
+      
+            unless @pids[pid][:request_finished]
+              @pids[pid][:last_memory_reading] = -1
+            end
+            @pids[pid][:action] = $1
+            @pids[pid][:request_finished] = false
+      
+          elsif line =~ /Memory usage: (\d+) /
+      
+            memory_reading = $1.to_i
+            @pids[pid][:current_memory_reading] = memory_reading
+      
+          elsif line =~ /Completed in/
+        
+            @pids[pid][:request_finished] = true
+            unless @pids[pid][:current_memory_reading] == -1 || @pids[pid][:last_memory_reading] == -1
+              memory_diff = @pids[pid][:current_memory_reading] - @pids[pid][:last_memory_reading]
+              if memory_diff > @threshold
+                @bad_actions[@pids[pid][:action]] ||= 0
+                @bad_actions[@pids[pid][:action]] = @bad_actions[@pids[pid][:action]] + 1
+                date = /^(\w+ \d{2} \d{2}:\d{2}:\d{2})/.match(line).captures[0]
+                @bad_requests.push(OinkedMemoryRequest.new(@pids[pid][:action], date, @pids[pid][:buffer], memory_diff))
+                if @format == :verbose
+                  @pids[pid][:buffer].each { |b| output.puts b } 
+                  output.puts "---------------------------------------------------------------------"
+                end
+              end
+            end
+          
+            @pids[pid][:buffer] = []
+            @pids[pid][:last_memory_reading] = @pids[pid][:current_memory_reading]
+            @pids[pid][:current_memory_reading] = -1
+        
+          end # end elsif
+        end # end each_line
+      end # end each input
+
+      print_summary(output)
+    
+    end
+
+  end
+  
+end

data/lib/oink/oinked_request/oinked_ar_request.rb

@@ -0,0 +1,9 @@
+require "oink/oinked_request/oinked_request"
+
+class OinkedARRequest < OinkedRequest
+  
+  def display_oink_number
+    @oink_number
+  end
+  
+end

data/lib/oink/oinked_request/oinked_memory_request.rb

@@ -0,0 +1,9 @@
+require "oink/oinked_request/oinked_request"
+
+class OinkedMemoryRequest < OinkedRequest
+  
+  def display_oink_number
+    "#{@oink_number} KB"
+  end
+  
+end

data/lib/oink/oinked_request/oinked_request.rb

@@ -0,0 +1,20 @@
+class OinkedRequest
+  
+  attr_accessor :action, :datetime, :log_lines, :oink_number
+  
+  def initialize(action, datetime, log_lines, oink_number)
+    @action = action
+    @datetime = datetime
+    @log_lines = log_lines
+    @oink_number = oink_number
+  end
+  
+  def <=>(other)
+    self.oink_number <=> other.oink_number
+  end
+  
+  def >(other)
+    self.oink_number > other.oink_number
+  end  
+  
+end

data/lib/oink/priority_queue.rb

@@ -0,0 +1,37 @@
+class PriorityQueue
+  
+  include Enumerable
+  
+  def initialize(size)
+    @size = size
+    @queue = []
+  end
+  
+  def push(item)
+    if @queue.size < @size
+      @queue << item
+    elsif item > @queue.last
+      @queue[@size - 1] = item
+    end
+    prioritize
+  end
+  
+  def to_a
+    @queue
+  end
+  
+  def size
+    @queue.size
+  end
+  
+  def each
+    @queue.each { |i| yield i }
+  end
+  
+protected
+
+  def prioritize
+    @queue.sort! { |a, b| b <=> a }
+  end
+  
+end

data/lib/oink/rails/instance_type_counter.rb

@@ -0,0 +1,84 @@
+module Oink
+  
+  def self.extended_active_record?
+    @oink_extended_active_record
+  end
+
+  def self.extended_active_record!
+    @oink_extended_active_record = true
+  end
+  
+  module InstanceTypeCounter
+    def self.included(klass)
+      ActiveRecord::Base.send(:include, OinkInstanceTypeCounterInstanceMethods)
+    
+      klass.class_eval do
+        after_filter :report_instance_type_count
+      end
+    end
+
+    def before_report_active_record_count(instantiation_data)
+    end
+
+    private
+    
+      def report_instance_type_count
+        report_hash = ActiveRecord::Base.instantiated_hash.merge("Total" => ActiveRecord::Base.total_objects_instantiated)
+        breakdown = report_hash.sort{|a,b| b[1]<=>a[1]}.collect {|k,v| "#{k}: #{v}" }.join(" | ")
+        before_report_active_record_count(breakdown)
+        if logger
+          logger.info("Instantiation Breakdown: #{breakdown}")
+        end
+        ActiveRecord::Base.reset_instance_type_count
+      end
+
+  end
+
+  module OinkInstanceTypeCounterInstanceMethods
+  
+    def self.included(klass)
+      klass.class_eval do
+      
+        @@instantiated = {}
+        @@total = nil
+      
+        def self.reset_instance_type_count
+          @@instantiated = {}
+          @@total = nil
+        end
+      
+        def self.instantiated_hash
+          @@instantiated
+        end
+        
+        def self.total_objects_instantiated
+          @@total ||= @@instantiated.inject(0) { |i, j| i + j.last }
+        end
+
+        unless Oink.extended_active_record?
+          if instance_methods.include?("after_initialize")
+            def after_initialize_with_oink
+              after_initialize_without_oink
+              increment_ar_count
+            end
+          
+            alias_method_chain :after_initialize, :oink
+          else
+            def after_initialize
+              increment_ar_count
+            end
+          end
+          
+          Oink.extended_active_record!
+        end
+      
+      end
+    end
+  
+    def increment_ar_count
+      @@instantiated[self.class.base_class.name] ||= 0
+      @@instantiated[self.class.base_class.name] = @@instantiated[self.class.base_class.name] + 1  
+    end
+
+  end
+end

data/lib/oink/rails/memory_usage_logger.rb

@@ -0,0 +1,17 @@
+module Oink
+  module MemoryUsageLogger
+    def self.included(klass)
+      klass.class_eval do
+        after_filter :log_memory_usage
+      end
+    end
+  
+    private
+      def log_memory_usage
+        if logger
+          memory_usage = `ps -o rss= -p #{$$}`.to_i
+          logger.info("Memory usage: #{memory_usage} | PID: #{$$}")
+        end
+      end
+  end
+end

metadata

@@ -0,0 +1,71 @@
+--- !ruby/object:Gem::Specification 
+name: aiwilliams-oink
+version: !ruby/object:Gem::Version 
+  version: 0.1.0
+platform: ruby
+authors: 
+- Noah Davis
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2009-05-22 00:00:00 -07:00
+default_executable: oink
+dependencies: []
+
+description: Log parser to identify actions which significantly increase VM heap size
+email: noahd1@yahoo.com
+executables: 
+- oink
+extensions: []
+
+extra_rdoc_files: []
+
+files: 
+- History.txt
+- MIT-LICENSE
+- README.rdoc
+- Rakefile
+- bin/oink
+- lib/oink
+- lib/oink/active_record_instantiation_reporter.rb
+- lib/oink/base.rb
+- lib/oink/cli.rb
+- lib/oink/memory_usage_reporter.rb
+- lib/oink/oinked_request
+- lib/oink/oinked_request/oinked_ar_request.rb
+- lib/oink/oinked_request/oinked_memory_request.rb
+- lib/oink/oinked_request/oinked_request.rb
+- lib/oink/priority_queue.rb
+- lib/oink/rails
+- lib/oink/rails/instance_type_counter.rb
+- lib/oink/rails/memory_usage_logger.rb
+- lib/oink.rb
+has_rdoc: false
+homepage: http://github.com/noahd1/oink
+post_install_message: 
+rdoc_options: []
+
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.2.0
+signing_key: 
+specification_version: 2
+summary: Log parser to identify actions which significantly increase VM heap size
+test_files: []
+