oink-transcode 0.1.2

data/History.txt

@@ -0,0 +1,4 @@
+=== 0.1.0 / 2009-02-09
+
+* 1st release
+

data/MIT-LICENSE

@@ -0,0 +1,21 @@
+Copyright (c) 2009 Noah Davis
+Copyright (c) 2008 Ben Johnson of Binary Logic (binarylogic.com)
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.rdoc

@@ -0,0 +1,136 @@
+=== Oink
+
+* http://github.com/noahd1/oink
+
+=== Description
+
+Rails plugin and log parser to help narrow down the source(s) of increased memory usage in rails applications.
+
+=== Synopsis
+
+Oink adds memory and active record instantiation information to rails log during runtime and provides an executable to help digest the enhanced logs.
+
+Given a minimum threshold and a metric (memory or active record instantiation), the oink executable reports:
+
+1. The top ten single requests which exceeded the threshold for the metric, ordered by the request which exceeded the threshold the most
+2. The number of times each action exceeded the threshold for the metric, ordered by the action which exceeded the threshold the most
+3. (in verbose mode) The log lines produced by requests which exceeded the threshold
+
+Many thanks to Ben Johnson for Memory Usage Logger (http://github.com/binarylogic/memory_usage_logger/tree/master) which is the basis of this plugin.
+
+== Dependencies
+
+Currently oink can only parse logs in the Hodel3000Logger format
+
+- http://github.com/topfunky/hodel_3000_compliant_logger
+
+When used as a gem, this is automatically brought in as a dependency.
+
+=== Installation and configuration
+
+To begin, you'll want to install it:
+
+  gem install oink
+
+Next, you'll need to update your config/environment.rb to include oink as a dependency, and to configure Rails to use the hodel_3000_compliant_logger: 
+
+  Rails::Initializer.run do |config|
+    # your other config here
+    config.gem 'oink'
+    begin
+      require 'hodel_3000_compliant_logger'
+      config.logger = Hodel3000CompliantLogger.new(config.log_path)
+    rescue LoadError => e
+      $stderr.puts "Hodel3000CompliantLogger unavailable, oink will be disabled"
+    end
+  end
+   
+
+Oink allows you to add two type of statistics to your log output: memory usage and number of ActiveRecord objects instantiated. Here's an example for the output you can get:
+
+  Feb 08 11:39:54 ey33-s00302 rails[9076]: Memory usage: 316516 | PID: 9076
+  Feb 27 13:21:04 ey04-s00295 rails[11862]: Instantiation Breakdown: Total: 73 | User: 34 | Group: 20 | Medium: 20 | Sport: 10 | Post: 4 | Discussion: 2
+
+For memory usage, add this to app/controllers/application_controller.rb: 
+
+  class ApplicationController
+    include Oink::MemoryUsageLogger
+  end
+
+For ActiveRecord objects instantiated, add this to your app/controllers/application_controller.rb:
+
+  class ApplicationController
+    include Oink::InstanceTypeCounter
+  end
+
+
+== Analyizing logs
+
+After installing the plugin and aggregating some enhanced logs, run the 'oink' executable against your server logs. 
+
+  Usage: oink [options] files
+      -t, --threshold [INTEGER]        Memory threshold in MB
+      -f, --file filepath              Output to file
+          --format FORMAT              Select format
+                                         (ss,v,s,verbose,short-summary,summary)
+      -m, --memory                     Check for Memory Threshold (default)
+      -r, --active-record              Check for Active Record Threshold
+
+Oink hunts for requests which exceed a given threshold. In "memory" mode (the default), the threshold represents a megabyte memory increase from the previous request. In "active record" mode (turned on by passing the --active-record switch), the threshold represents the number of active record objects instantiated during a request.
+
+e.g. To find all actions which increase the heap size more than 75 MB, where log files are location in /tmp/logs/
+
+  $ oink --threshold=75 /tmp/logs/*
+  ---- MEMORY THRESHOLD ----
+  THRESHOLD: 75 MB
+
+  -- SUMMARY --
+  Worst Requests:
+  1. Feb 02 16:26:06, 157524 KB, SportsController#show
+  2. Feb 02 20:11:54, 134972 KB, DashboardsController#show
+  3. Feb 02 19:06:13, 131912 KB, DashboardsController#show
+  4. Feb 02 08:07:46, 115448 KB, GroupsController#show
+  5. Feb 02 12:19:53, 112924 KB, GroupsController#show
+  6. Feb 02 13:03:00, 112064 KB, ColorSchemesController#show
+  7. Feb 02 13:01:59, 109148 KB, SessionsController#create
+  8. Feb 02 06:11:17, 108456 KB, PublicPagesController#join
+  9. Feb 02 08:43:06, 94468 KB, CommentsController#create
+  10. Feb 02 20:49:44, 82340 KB, DashboardsController#show
+
+  Worst Actions:
+  10, DashboardsController#show
+  9, GroupsController#show
+  5, PublicPagesController#show
+  5, UsersController#show
+  3, MediaController#show
+  2, SportsController#show
+  1, SessionsController#create
+  1, GroupInvitesController#by_email
+  1, MediaController#index
+  1, PostsController#show
+  1, PhotoVotesController#create
+  1, AlbumsController#index
+  1, SignupsController#new
+  1, ColorSchemesController#show
+  1, PublicPagesController#join
+  1, CommentsController#create
+
+e.g. In verbose mode, oink will print out all the log information from your logs about the actions which exceeded the threshold specified
+
+  $ oink --format verbose --threshold=75 /tmp/logs/*
+
+  ---------------------------------------------------------------------
+  Feb 08 11:39:52 ey33-s00302 rails[9076]: Processing UsersController#show (for 11.187.34.45 at 2009-02-08 11:39:52) [GET]
+  Feb 08 11:39:52 ey33-s00302 rails[9076]: Parameters: {"action"=>"show", "id"=>"45", "controller"=>"users"}
+  Feb 08 11:39:52 ey33-s00302 rails[9076]: Rendering template within layouts/application
+  Feb 08 11:39:52 ey33-s00302 rails[9076]: Rendering users/show
+  Feb 08 11:39:54 ey33-s00302 rails[9076]: Memory usage: 316516 | PID: 9076
+  Feb 08 11:39:54 ey33-s00302 rails[9076]: Completed in 2008ms (View: 1136, DB: 264) | 200 OK [http://www.example.com/users/45]
+  ---------------------------------------------------------------------
+
+Verbose format prints the summary as well as each action which exceeded the threshold.
+
+=== Authors
+
+- Maintained by Noah Davis
+- Thanks to Weplay (http://weplay.com) for sponsoring development and supporting open sourcing it from the start

data/Rakefile

@@ -0,0 +1,33 @@
+require 'rubygems'
+require "rake/gempackagetask"
+require "rake/clean"
+require "spec/rake/spectask"
+require './lib/oink/base.rb'
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |s|
+    s.name         = "oink-transcode"
+    s.version      = Oink::Base::VERSION
+    s.author       = ["Noah Davis", "Shane Gibbons"]
+    s.email        = "shane.r.gibbons" + "@" + "gmail.com"
+    s.homepage     = "http://github.com/microsage/oink"
+    s.summary      = "Log parser to identify actions which significantly increase VM heap size"
+    s.description  = s.summary
+    s.executables  = "oink"
+    s.files        = %w[History.txt MIT-LICENSE README.rdoc Rakefile] + Dir["bin/*"] + Dir["lib/**/*"]
+    s.add_dependency 'hodel_3000_compliant_logger'
+  end
+  Jeweler::GemcutterTasks.new
+rescue LoadError
+  puts "Jeweler (or a dependency) not available. Install it with: gem install jeweler"
+end
+
+Spec::Rake::SpecTask.new do |t|
+  t.spec_opts == ["--color"]
+end
+
+desc "Run the specs"
+task :default => ["spec"]
+
+CLEAN.include ["pkg", "*.gem", "doc", "ri", "coverage"]

data/bin/oink

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+
+require File.dirname(__FILE__) + "/../lib/oink.rb"
+Cli.new(ARGV.dup).process
+

data/lib/oink.rb

@@ -0,0 +1,9 @@
+$:.unshift(File.dirname(__FILE__ + '.rb') + '/../lib') unless $:.include?(File.dirname(__FILE__ + '.rb') + '/../lib')
+
+require "oink/memory_usage_reporter"
+require "oink/active_record_instantiation_reporter"
+require "oink/cli"
+
+if defined?(Rails)
+  require 'oink/rails'
+end

data/lib/oink/active_record_instantiation_reporter.rb

@@ -0,0 +1,68 @@
+require "date"
+require "oink/base"
+require "oink/oinked_request/oinked_ar_request"
+require "oink/priority_queue"
+
+module Oink
+
+  class ActiveRecordInstantiationReporter < Base
+  
+    def print(output)
+      output.puts "---- OINK FOR ACTIVERECORD ----"
+      output.puts "THRESHOLD: #{@threshold} Active Record objects per request\n"
+    
+      output.puts "\n-- REQUESTS --\n" if @format == :verbose
+    
+      @inputs.each do |input|
+        input.each_line do |line|
+          line = line.strip
+          
+           # Skip this line since we're only interested in the Hodel 3000 compliant lines
+          next unless line =~ HODEL_LOG_FORMAT_REGEX
+
+          if line =~ /rails\[(\d+)\]/
+            pid = $1
+            @pids[pid] ||= { :buffer => [], :ar_count => -1, :action => "", :request_finished => true }
+            @pids[pid][:buffer] << line
+          end
+
+          if line =~ /Processing ((\w+)#(\w+)) /
+
+            @pids[pid][:action] = $1
+            unless @pids[pid][:request_finished]
+              @pids[pid][:buffer] = [line]
+            end
+            @pids[pid][:request_finished] = false
+      
+          elsif line =~ /Instantiation Breakdown: Total: (\d+)/
+
+            @pids[pid][:ar_count] = $1.to_i
+      
+          elsif line =~ /Completed in/
+
+            if @pids[pid][:ar_count] > @threshold
+              @bad_actions[@pids[pid][:action]] ||= 0
+              @bad_actions[@pids[pid][:action]] = @bad_actions[@pids[pid][:action]] + 1
+              date = HODEL_LOG_FORMAT_REGEX.match(line).captures[0]
+              @bad_requests.push(OinkedARRequest.new(@pids[pid][:action], date, @pids[pid][:buffer], @pids[pid][:ar_count]))
+              if @format == :verbose
+                @pids[pid][:buffer].each { |b| output.puts b } 
+                output.puts "---------------------------------------------------------------------"
+              end
+            end
+          
+            @pids[pid][:request_finished] = true
+            @pids[pid][:buffer] = []
+            @pids[pid][:ar_count] = -1
+        
+          end # end elsif
+        end # end each_line
+      end # end each input
+
+      print_summary(output)
+    
+    end
+
+  end
+  
+end

data/lib/oink/base.rb

@@ -0,0 +1,40 @@
+module Oink
+
+  class Base
+
+    VERSION = '0.1.2'
+    FORMATS = %w[verbose short-summary summary]
+    FORMAT_ALIASES = { "v" => "verbose", "ss" => "short-summary", "s" => "summary" }
+    HODEL_LOG_FORMAT_REGEX = /^(\w+ \d{2} \d{2}:\d{2}:\d{2})/
+
+    def initialize(input, threshold, options = {})
+      @inputs = Array(input)
+      @threshold = threshold
+      @format = options[:format] || :short_summary
+  
+      @pids = {}
+      @bad_actions = {}
+      @bad_requests = PriorityQueue.new(10)
+    end
+
+  protected
+
+    def print_summary(output)
+      output.puts "\n-- SUMMARY --\n"
+      output.puts "Worst Requests:"
+      @bad_requests.each_with_index do |offender, index|
+        output.puts "#{index + 1}. #{offender.datetime}, #{offender.display_oink_number}, #{offender.action}"
+        if @format == :summary
+          offender.log_lines.each { |b| output.puts b } 
+          output.puts "---------------------------------------------------------------------"
+        end
+      end
+      output.puts "\nWorst Actions:"
+      @bad_actions.sort{|a,b| b[1]<=>a[1]}.each { |elem|
+        output.puts "#{elem[1]}, #{elem[0]}"
+      }
+    end
+
+  end
+  
+end

data/lib/oink/cli.rb

@@ -0,0 +1,97 @@
+require 'optparse'
+
+class Cli
+  
+  def initialize(args)
+    @args = args
+  end
+  
+  def process
+    options = { :format => :short_summary, :type => :memory }
+
+    op = OptionParser.new do |opts|
+      opts.banner = "Usage: oink [options] files"
+
+      opts.on("-t", "--threshold [INTEGER]", Integer,
+              "Memory threshold in MB") do |threshold|
+        options[:threshold] = threshold
+      end
+
+      opts.on("-f", "--file filepath", "Output to file") do |filename|
+        options[:output_file] = filename
+      end
+
+      format_list = (Oink::MemoryUsageReporter::FORMAT_ALIASES.keys + Oink::MemoryUsageReporter::FORMATS).join(',')
+      opts.on("--format FORMAT", Oink::MemoryUsageReporter::FORMATS, Oink::MemoryUsageReporter::FORMAT_ALIASES, "Select format",
+              "  (#{format_list})") do |format|
+        options[:format] = format.to_sym
+      end
+
+      opts.on("-m", "--memory", "Check for Memory Threshold (default)") do |v|
+         options[:type] = :memory
+      end
+
+      opts.on("-r", "--active-record", "Check for Active Record Threshold") do |v|
+         options[:type] = :active_record
+      end
+
+    end
+
+    op.parse!(@args)
+
+    if @args.empty?
+      puts op
+      exit
+    end
+
+    output = nil
+
+    if options[:output_file]
+      output = File.open(options[:output_file], 'w')
+    else
+      output = STDOUT
+    end
+
+    files = get_file_listing(@args)
+
+    handles = files.map { |f| File.open(f) }
+
+    if options[:type] == :memory
+
+      options[:threshold] ||= 75
+      options[:threshold] *= 1024
+
+      Oink::MemoryUsageReporter.new(handles, options[:threshold], :format => options[:format]).print(output)
+
+    elsif options[:type] == :active_record
+
+      options[:threshold] ||= 500
+
+      Oink::ActiveRecordInstantiationReporter.new(handles, options[:threshold], :format => options[:format]).print(output)
+
+    end
+
+    output.close
+    handles.each { |h| h.close }    
+  end
+  
+protected
+
+  def get_file_listing(args)
+    listing = []
+    args.each do |file|
+      unless File.exist?(file)
+        raise "Could not find \"#{file}\""
+      end
+      if File.directory?(file)
+        listing += Dir.glob("#{file}/**")
+      else
+        listing << file
+      end
+    end
+    listing
+  end
+
+end
+
+

data/lib/oink/memory_usage_reporter.rb

@@ -0,0 +1,74 @@
+require "date"
+require "oink/base"
+require "oink/oinked_request/oinked_memory_request"
+require "oink/priority_queue"
+require "iconv"
+
+module Oink
+
+  class MemoryUsageReporter < Base
+    def print(output)
+      output.puts "---- MEMORY THRESHOLD ----"
+      output.puts "THRESHOLD: #{@threshold/1024} MB\n"
+    
+      output.puts "\n-- REQUESTS --\n" if @format == :verbose
+    
+      ic = Iconv.new('UTF-8//IGNORE', 'UTF-8')
+      @inputs.each do |input|
+        input.each_line do |line|
+          line = ic.iconv(line.strip + ' ')[0..-2]
+          
+           # Skip this line since we're only interested in the Hodel 3000 compliant lines
+          next unless line =~ HODEL_LOG_FORMAT_REGEX
+
+          if line =~ /rails\[(\d+)\]/
+            pid = $1
+            @pids[pid] ||= { :buffer => [], :last_memory_reading => -1, :current_memory_reading => -1, :action => "", :request_finished => true }
+            @pids[pid][:buffer] << line
+          end
+
+          if line =~ /Processing ((\w+)#(\w+)) /
+      
+            unless @pids[pid][:request_finished]
+              @pids[pid][:last_memory_reading] = -1
+            end
+            @pids[pid][:action] = $1
+            @pids[pid][:request_finished] = false
+      
+          elsif line =~ /Memory usage: (\d+) /
+      
+            memory_reading = $1.to_i
+            @pids[pid][:current_memory_reading] = memory_reading
+      
+          elsif line =~ /Completed in/
+        
+            @pids[pid][:request_finished] = true
+            unless @pids[pid][:current_memory_reading] == -1 || @pids[pid][:last_memory_reading] == -1
+              memory_diff = @pids[pid][:current_memory_reading] - @pids[pid][:last_memory_reading]
+              if memory_diff > @threshold
+                @bad_actions[@pids[pid][:action]] ||= 0
+                @bad_actions[@pids[pid][:action]] = @bad_actions[@pids[pid][:action]] + 1
+                date = HODEL_LOG_FORMAT_REGEX.match(line).captures[0]
+                @bad_requests.push(OinkedMemoryRequest.new(@pids[pid][:action], date, @pids[pid][:buffer], memory_diff))
+                if @format == :verbose
+                  @pids[pid][:buffer].each { |b| output.puts b } 
+                  output.puts "---------------------------------------------------------------------"
+                end
+              end
+            end
+          
+            @pids[pid][:buffer] = []
+            @pids[pid][:last_memory_reading] = @pids[pid][:current_memory_reading]
+            @pids[pid][:current_memory_reading] = -1
+        
+          end # end elsif
+        end # end each_line
+      end # end each input
+
+      print_summary(output)
+    
+    end
+
+  end
+  
+end