bleak_house 4.6 → 5

data/CHANGELOG

@@ -1,63 +1,10 @@
 
-v4.6. Allow snapshotting without running the GC first.
-
-v4.5. Enable SET_CURRENT_SOURCE() for better reporting (tmm1, joedamato).
-
-v4.4. Switch to Ruby 1.8.7p174.
-
-v4.3. Build against OpenSSL 0.9.8j or later (kennethkalmer). Bugfixes (Pete Hodgson, joshuabates).
-
-v4.2.1. Harden patch/build process. Fix bug.
-
-v4.1.1. Update bundled Ruby to patchlevel 230.
-
-v4.1. Support calculating deltas in analyze utility.
-
-v4.0. Track spawn points live instead of sampling.
-
-v3.7.1. Fix some nil issues with corrupted dumps (cody caughlan).
-
-v3.7. Sample object contents. Restore Rails 1.2.x compatibility.
-
-v3.6. Rails 2.0.2 compatibility.
-
-v3.5.1. Update bundled Ruby to patchlevel 110.
-
-v3.5. Explain when you need more frames. Use a cross-platform version of 'which'. Slight accuracy improvement.
-
-v3.4. Clearer output descriptions; work around a Marshal bug on x64; fix for missing immortal leaks.
-
-v3.3. Build Ruby in gem install step; bundle Ruby 1.8.6 source; fixes for truncated final frames.
-
-v3.2. Use Ccsv for faster parsing.
-
-v3.0.3. Caching; build task; impact factor.
-
-v3.0.1. Rewrite. Remove Gruffs and graphs. Become a standard compiled extension. Walk the symbol table. Track object histories. Use custom CSV format instead of YAML.
-
-v2.7. Version renumbering; apologies if this messes anyone up.
-
-v2.6. RDoc documentation; use new Echoe setup; fix for namespaced controllers.
-
-v2.5. Detect if core keys are present or not; fixes for analyzing non-Rails apps.
-
-v2.4. Log heap usage; remove pure-Ruby profiler.
-
-v2.3. Subtract Rails core counts from action counts and join paired frames.
-
-v2.2. Change smoothing algorithm to be more intuitive and so mem usage is correct.
-
-v2.1. Close file properly.
-
-v2. C instrumentation with YAML output.
-
-v1.3. Log memory usage on *nix.
-
-v1.2. Fix nil directory traversal error on Windows.
-
-v1.1. Gem version
-
-v1. Dumped Rublique; replaced with a lightweight non-delta marshalling version.
-
-v0. First version
+BleakHouse Changelog
+
+5. totally awesome C instrumentation
+4. log memory usage on *nix
+3.1. fix nil directory traversal error on windows
+3. gem version
+2. dumped rublique once I understood what it did; replaced with a lightweight non-delta marshalling version
+1. first version
 

data/Manifest

@@ -1,21 +1,22 @@
-CHANGELOG
-LICENSE
-LICENSE_BSD
-Manifest
-README
-Rakefile
-TODO
-bin/bleak
-ext/build_ruby.rb
-ext/build_snapshot.rb
-ext/extconf.rb
-ext/snapshot.c
-ext/snapshot.h
-lib/bleak_house.rb
-lib/bleak_house/analyzer.rb
-lib/bleak_house/hook.rb
-ruby/ruby-1.8.7-p174.tar.bz2
-ruby/ruby-1.8.7.patch
-test/benchmark/bench.rb
-test/test_helper.rb
-test/unit/test_bleak_house.rb
+./CHANGELOG
+./LICENSE_AFL
+./Manifest
+./README
+./Rakefile
+./init.rb
+./install.rb
+./lib/bleak_house/LICENSE_BSD
+./lib/bleak_house/action_controller.rb
+./lib/bleak_house/analyze.rb
+./lib/bleak_house/bleak_house.rb
+./lib/bleak_house/c.rb
+./lib/bleak_house/dispatcher.rb
+./lib/bleak_house/gruff_hacks.rb
+./lib/bleak_house/mem_logger.rb
+./lib/bleak_house/rake_task_redefine_task.rb
+./lib/bleak_house/ruby.rb
+./lib/bleak_house/support_methods.rb
+./lib/bleak_house.rb
+./patches/gc.c.patch
+./tasks/bleak_house_tasks.rake
+./test/unit/test_bleak_house.rb

data/README

@@ -1,125 +1,41 @@
 
 BleakHouse
 
-A library for finding memory leaks. Patched for Ruby 1.8.7p72. 
+REQUIREMENTS:
 
-An experiment.
-Run git clone and then rake gem;
-gem install bleak_house
+A *nix operating system. Windows is not supported.
 
-== License
+Gems: 'gruff', 'rmagick', 'active_support', 'RubyInline'
 
-Copyright 2007, 2008 Cloudburst, LLC. Licensed under the AFL 3. See the included LICENSE file. Portions copyright 2006 Eric Hodel and used with permission. See the included LICENSE_BSD file.
+It is highly recommended that you compile the bleak_house patched 
+Ruby build. In the plugin directory: 
+  rake ruby:build
 
-The public certificate for this gem is here[http://rubyforge.org/frs/download.php/25331/evan_weaver-original-public_cert.pem].
+USAGE:
 
-If you use this software, please {make a donation}[http://blog.evanweaver.com/donate/], or {recommend Evan}[http://www.workingwithrails.com/person/7739-evan-weaver] at Working with Rails.
+To profile your application:
+  RAILS_ENV=production BLEAK_HOUSE=true ruby-bleak-house ./script/server 
+  
+Browse around manually, thrash your entire app with a script, 
+target troublesome controllers/actions, etc.
 
-== Features
+Then, to analyze:
+  RAILS_ENV=production SMOOTHNESS=2 rake bleak_house:analyze
 
-* leak-proof C instrumentation
-* minimal impact on runtime performance
-* fast analysis step
-* tracks all objects allocated on the heap, including internal types like <tt>T_NODE</tt>
-* easy integration into any program, not just Rails
+And then look in log/bleak_house/.
 
-== Requirements
+TROUBLESHOOTING
 
-* A unix-like operating system
-* Ruby 1.8.7
+If you see the error "Symbol not found: _rb_gc_heaps_used",
+it means you installed the patched binary, but tried to run the 
+server with the regular binary.
 
-= Usage
+COPYRIGHT AND LICENSING
 
-== Installation
+Copyright (c) 2007 Cloudburst, LLC. See the included LICENSE_AFL 
+file.
 
-Install the gem:
-  sudo gem install bleak_house
-
-The installation takes a long time because it compiles a patched Ruby 1.8.7 binary for you. It is installed as <tt>ruby-bleak-house</tt> alongside your regular <tt>ruby</tt> binary.
-
-Please see the forum ( http://rubyforge.org/forum/forum.php?forum_id=13983 ) if you have installation problems.
-
-== Usage
-
-We will profile a Rails app as an example. Note that BleakHouse works equally well in any Ruby program.
-
-First, to setup the app for profiling, add the following at the bottom of <tt>config/environment.rb</tt>:
-  require 'bleak_house' if ENV['BLEAK_HOUSE']
-
-Then, to engage the logger (possibly in a live deployment situation), start a server instance as so:
-  RAILS_ENV=production BLEAK_HOUSE=1 ruby-bleak-house ./script/server
-
-Look for the message:
-  ** Bleakhouse: installed
-
-Exercise your app. After a couple hundred requests, hit CTRL-C. The server will stop and BleakHouse will produce a dumpfile in <tt>/tmp</tt>:
-
-  ** BleakHouse: working...
-  ** BleakHouse: complete
-  ** Bleakhouse: run 'bleak /tmp/bleak.5979.000.dump' to analyze.
-
-To analyze it, just run the listed command. The top 20 leakiest lines will be listed:
-
-  191691 total objects
-  Final heap size 191691 filled, 220961 free
-  Displaying top 20 most common line/class pairs
-  89513 __null__:__null__:__node__
-  41438 __null__:__null__:String
-  2348 /opt/local//lib/ruby/site_ruby/1.8/rubygems/specification.rb:557:Array
-  1508 /opt/local//lib/ruby/gems/1.8/specifications/gettext-1.90.0.gemspec:14:String
-  1021 /opt/local//lib/ruby/gems/1.8/specifications/heel-0.2.0.gemspec:14:String
-   951 /opt/local//lib/ruby/site_ruby/1.8/rubygems/version.rb:111:String
-   935 /opt/local//lib/ruby/site_ruby/1.8/rubygems/specification.rb:557:String
-   834 /opt/local//lib/ruby/site_ruby/1.8/rubygems/version.rb:146:Array
-  ...
-
-You can pass an integer as the second parameter to <tt>bleak</tt> if you want to see more lines than the default.
-
-The underscored types are special Ruby internal structs, but can be real leaks just as easily as fullblown classes.
-
-= Extras
-
-== Injecting a signal
-
-You can send <tt>SIGUSR2</tt> to a BleakHouse-instrumented program to snag a dump at any time. Once the dump completes, the program will continue to run. Dumps are named based on the host process id, and sequential dumps are numbered in ascending order.
-
-== Tips
-
-Do not try to detect Rails leaks in <tt>development</tt> mode. Make a separate <tt>benchmark</tt> environment if you need to, and make sure all your production caching is turned on.
-
-It is normal to see lots of <tt>null:null</tt> references, especially for nodes. Using <tt>eval()</tt> too much can be a cause of node leaks. You can sometimes track <tt>eval()</tt> by using sourceline macros in your code:
-
-  eval("CODE", nil, __FILE__, __LINE__)
-
-You may get library require errors if you install <tt>ruby-bleak-house</tt> 1.8.7 alongside a different verson of Ruby. You could try to patch your local version of Ruby instead, or you can get <tt>ruby-bleak-house</tt> to lie about its version. Just make sure that the <tt>bleak-house</tt> library is the first thing required (even before Rubygems):
-
-  ruby-bleak-house -I `ruby -e 'puts \`gem which bleak_house\`.split("\n")[1][0..-16]'` -rbleak_house
-
-It is not recommended that you use <tt>ruby-bleak-house</tt> as your production Ruby binary, since it will be slightly slower and use slightly more memory. It is unlikely, however, to affect stability.
-
-If BleakHouse doesn't report any heap growth but you still have memory growth, you might have a broken C extension, or have encounted a {real leak in the interpreter}[http://groups.google.com/group/god-rb/browse_thread/thread/01cca2b7c4a581c2]. Try using Valgrind[http://blog.evanweaver.com/articles/2008/02/05/valgrind-and-ruby/] instead.
-
-== Methods
-
-The easiest way to fix a leak is to make it repeatable.
-
-First, write a script that exercises your app in a deterministic way. Run it for a small number of loops; then run <tt>bleak</tt>. Then run it for a larger number of loops, and run <tt>bleak</tt> again. The lines that grow significantly between runs are your leaks for that codepath.
-
-Now, look at those lines in the source and try to figure out what references them. Where do the return values go? Add some breakpoints or output backtraces to <tt>STDERR</tt> as you go. Eventually you should find a point where it is relatively clear that a reference is getting maintained.
-
-Try to remove that reference, run your script again, and see if the object counts have dropped.
-
-== Reporting problems
-
-The support forum is here[http://rubyforge.org/forum/forum.php?forum_id=13983].
-
-Patches and contributions are very welcome. Please note that contributors are required to assign copyright for their additions to Cloudburst, LLC.
-
-== Further resources
-
-* http://blog.evanweaver.com/articles/2008/04/06/bleakhouse-4/
-* http://blog.evanweaver.com/articles/2008/02/05/valgrind-and-ruby/
-* http://blog.evanweaver.com/articles/2007/05/12/let-me-hit-you-with-some-knowledge
-* http://blog.evanweaver.com/articles/2007/05/06/leak-proof-direct-heap-instrumentation-for-bleak_house
-* http://blog.evanweaver.com/articles/2007/04/28/bleak_house
+Portions of lib/bleak_house/c.rb copyright (c) 2006 Eric Hodel and 
+used under license (see included the lib/bleak_house/LICENSE_BSD 
+file).
 

data/Rakefile

@@ -1,12 +1,104 @@
+require 'rubygems'
+require 'rake'
+require 'lib/bleak_house/rake_task_redefine_task'
 
-require 'echoe'
+DIR = File.dirname(__FILE__)
 
-Echoe.new("bleak_house") do |p|
-  p.author = "Evan Weaver"
-  p.project = "fauna"
-  p.summary = "A library for finding memory leaks."
-  p.url = "http://blog.evanweaver.com/files/doc/fauna/bleak_house/"
-  p.docs_host = 'blog.evanweaver.com:~/www/bax/public/files/doc/'
-  p.require_signed = true
-  p.rdoc_pattern = /^ext.*\.c|lib.*|^README|^CHANGELOG|^TODO|^LICENSE|^COPYING$/
+begin
+  require 'rake/clean'
+  gem 'echoe', '>= 1.2'
+  require 'echoe'
+  require 'fileutils'
+  include FileUtils
+
+  CLEAN.include ['**/.*.sw?', '*.gem', '.config']  
+  VERS = `cat CHANGELOG`[/^([\d\.]+)\. /, 1]
+  
+  taskmsg = File.open(DIR + "/tasks/bleak_house_tasks.rake").readlines
+  taskmsg = taskmsg[0..3] + [taskmsg[7][2..-1]] + taskmsg[9..-1]
+  
+  echoe = Echoe.new("bleak_house", VERS) do |p|
+    p.author = "Evan Weaver" 
+    p.rubyforge_name = "fauna"
+    p.name = "bleak_house"
+    p.description = "BleakHouse is a Rails plugin for finding memory leaks. It tracks ObjectSpace for your entire app, and produces charts of references by controller, by action, and by object class."
+    p.changes = `cat CHANGELOG`[/^([\d\.]+\. .*)/, 1]
+    p.email = "evan at cloudbur dot st"
+    p.summary = p.description
+    p.url = "http://blog.evanweaver.com"
+    p.need_tar = false
+    p.need_tar_gz = true
+    p.test_globs = ["*_test.rb"]
+    p.extra_deps = ["RubyInline"]
+    p.clean_globs = CLEAN  
+    p.spec_extras = {:post_install_message => 
+"
+Thanks for installing Bleak House #{VERS}. 
+
+For each Rails app you want to profile, you will need to add the following 
+rake task in RAILS_ROOT/lib/tasks/bleak_house_tasks.rake to be able to run 
+the analyzer: 
+" + taskmsg.join("  ") + "\n"}
+  end
+            
+rescue LoadError => boom
+  puts "You are missing a dependency required for meta-operations on this gem."
+  puts "#{boom.to_s.capitalize}."
+  
+  desc 'Run tests.'
+  task :default => :test
+end
+
+desc 'Run tests.'
+Rake::Task.redefine_task("test") do
+  system "ruby-bleak-house -Ibin:lib:test test/unit/test_bleak_house.rb #{ENV['METHOD'] ? "--name=#{ENV['METHOD']}" : ""}"
 end
+
+desc 'Build a patched binary.'
+namespace :ruby do
+  task :build do    
+    if RUBY_PLATFORM =~ /win32|windows/
+      puts "ERROR: windows not supported."
+      exit
+    end
+    require 'fileutils'
+    puts "building patched Ruby binary"
+    tmp = "/tmp/"
+    Dir.chdir(tmp) do
+      build_dir = "bleak_house"
+      binary_dir = File.dirname(`which ruby`)
+      FileUtils.rm_rf(build_dir) rescue nil
+      Dir.mkdir(build_dir)
+      begin
+        Dir.chdir(build_dir) do
+          puts "  downloading Ruby source"
+          bz2 = "ruby-1.8.6.tar.bz2"
+          system("wget 'http://rubyforge.org/frs/download.php/20425/ruby-1.8.6.tar.bz2' &> wget.log")
+          puts "  extracting"
+          system("tar xjf #{bz2} &> tar.log")
+          File.delete bz2
+          Dir.chdir("ruby-1.8.6") do
+            puts "  patching" or system("patch -p0 < \'#{DIR}/patches/gc.c.patch\' &> ../patch.log")
+            puts "  configuring" or system("./configure --prefix=#{binary_dir[0..-5]} &> ../configure.log") # --with-static-linked-ext
+            puts "  making" or system("make &> ../make.log")
+  
+            binary = "#{binary_dir}/ruby-bleak-house"
+            puts "  installing as #{binary}"
+            FileUtils.cp("./ruby", binary)
+            FileUtils.chmod(0755, binary)
+            puts "  done"
+          end
+        end
+      rescue Object => e
+        puts "ERROR: please see the last modified log file in #{tmp}#{build_dir}, perhaps\nit will contain a clue."
+      end    
+    end    
+  end
+end
+
+#def check_configure_size
+#  if (size = File.size("configure")) != 476155
+#    raise "Configure size is wrong (got #{size})"
+#  end
+#end
+

data/init.rb

@@ -0,0 +1,2 @@
+
+require 'bleak_house'

data/install.rb

@@ -0,0 +1,7 @@
+puts "
+
+I was such a shy little thing that I seldom dared to open my lips, 
+and never dared to open my heart, to anybody else.
+- Dickens, Bleak House
+
+"

data/lib/bleak_house.rb

@@ -1,22 +1,13 @@
 
-unless RUBY_PATCHLEVEL >= 905
-  raise "This build of Ruby has not been successfully patched for BleakHouse."
-end
-
-RUBY_VERSION = `ruby -v`.split(" ")[1]
-require 'snapshot'
-require 'bleak_house/hook'
+if ENV['BLEAK_HOUSE']
 
-class << BleakHouse
-  private :ext_snapshot
-end
+  require 'dispatcher' # rails  
   
-module BleakHouse
+  require 'bleak_house/bleak_house'
+  require 'bleak_house/mem_logger'
+  require 'bleak_house/dispatcher'
+  require 'bleak_house/action_controller'
 
-  # Walk the live, instrumented objects on the heap and write them to 
-  # <tt>logfile</tt>. Accepts an optional number of GC runs to perform
-  # before dumping the heap.
-  def self.snapshot(logfile, gc_runs = 3)
-    ext_snapshot(logfile, gc_runs)
-  end
+  BleakHouse.warn "enabled (log/bleak_house_#{RAILS_ENV}.dump)"
+    
 end

data/lib/bleak_house/action_controller.rb

@@ -0,0 +1,16 @@
+
+class ActionController::Base
+  class << self
+    def process_with_bleak_house(request, *args) 
+      BleakHouse.set_request_name request
+      process_without_bleak_house(request, *args)
+    end
+    alias_method_chain :process, :bleak_house
+    
+    def process_with_exception_with_bleak_house(request, *args)
+      BleakHouse.set_request_name request, "/error"
+      process_with_exception_without_bleak_house(request, *args)
+    end
+    alias_method_chain :process_with_exception, :bleak_house    
+  end
+end

data/lib/bleak_house/analyze.rb

@@ -0,0 +1,135 @@
+
+require 'rubygems'
+require 'fileutils'
+require 'yaml'
+require 'active_support'
+
+gem 'gruff', '= 0.2.8'
+require 'gruff'
+
+# require, but make rdoc not whine
+load "#{File.dirname(__FILE__)}/gruff_hacks.rb"
+load "#{File.dirname(__FILE__)}/support_methods.rb"
+
+Gruff::Base::LEFT_MARGIN = 200
+Gruff::Base::NEGATIVE_TOP_MARGIN = 30
+Gruff::Base::MAX_LEGENDS = 28
+
+class BleakHouse
+  class Analyze    
+  
+    SMOOTHNESS = ((i = ENV['SMOOTHNESS'].to_i) < 2 ? 2 : i)/2 * 2
+    MEM_KEY = "memory usage"
+    DIR = "#{RAILS_ROOT}/log/bleak_house/"
+    
+    def initialize(data, increments, name)
+      @data = data
+      @increments = increments
+      @name = name
+    end 
+    
+    def draw    
+      g = Gruff::Line.new("1024x768")
+      g.title = @name
+      g.x_axis_label = "time"
+      g.legend_font_size = g.legend_box_size = 14
+      g.title_font_size = 24
+      g.marker_font_size = 14
+            
+      @data.map do |key, values|        
+        ["#{(key.to_s.empty? ? '[Unknown]' : key).gsub(/.*::/, '')} (#{key == MEM_KEY ? "relative" : values.to_i.max})", values] # hax
+      end.sort_by do |key, values|
+        0 - key[/.*?([\d]+)\)$/, 1].to_i
+      end.each do |key, values|
+        g.data(key, values.to_i)
+      end
+      
+      labels = {}
+      mod = (@increments.size / 4.0).ceil
+      @increments.each_with_index do |increment, index|
+        labels[index] = increment if (index % mod).zero?
+      end
+      g.labels = labels
+      
+      g.minimum_value = 0
+#      g.maximum_value = @maximum
+      
+      g.write(@name.to_filename + ".png")
+    end
+    
+    def self.aggregate(data, selector, namer)
+      aggregate_data = {}
+      increments = []
+      data.each_with_index do |frameset, index|
+        increments << frameset.time
+        frameset.data.keys.select do |key| 
+          key =~ selector #or key =~ Regexp.new(specials.keys.join('|'))
+        end.each do |key|
+          aggregate_data[key.to_s[namer, 1]] ||= []
+          aggregate_data[key.to_s[namer, 1]][index] += frameset.data[key].to_i
+        end
+      end                 
+      [aggregate_data, increments]
+    end
+    
+    def self.build_all(filename)
+      unless File.exists? filename
+        puts "No data file found: #{filename}"
+        exit 
+      end
+      FileUtils.rm_r(DIR) rescue nil
+      Dir.mkdir(DIR)
+
+      Dir.chdir(DIR) do        
+
+        puts "parsing data"
+        data = YAML.load_file(filename)
+        data = data[0..(-1 - data.size % SMOOTHNESS)]
+        data = data.in_groups_of(SMOOTHNESS).map do |frames|
+          timestamp = frames.map(&:time).sum / SMOOTHNESS
+          values = frames.map(&:data).inject(Hash.new(0)) do |total, this_frame|
+            this_frame.each do |key, value|
+              total[key] += value
+            end
+            total
+          end
+          [Time.at(timestamp).strftime("%H:%M:%S"), values]
+        end                     
+        puts "#{data.size} frames after smoothing"
+
+        puts "entire app"
+        controller_data, increments = aggregate(data, //, /^(.*?)($|\/|::::)/)
+        if controller_data.has_key? MEM_KEY
+          controller_data_without_memory = controller_data.dup
+          controller_data_without_memory.delete(MEM_KEY)
+          scale_factor = controller_data_without_memory.values.flatten.to_i.max / controller_data[MEM_KEY].max.to_f * 0.8 rescue 1
+          controller_data[MEM_KEY] = controller_data[MEM_KEY].map{|x| (x * scale_factor).to_i }
+        end
+        Analyze.new(controller_data, increments, "objects by controller").draw
+                
+        # in each controller, by action
+        controller_data.keys.each do |controller|
+          @mem = (controller == MEM_KEY)
+          puts(@mem ? "  #{controller}" : "  action for #{controller} controller")
+          Dir.descend(controller) do             
+            action_data, increments = aggregate(data, /^#{controller}($|\/|::::)/, /\/(.*?)($|\/|::::)/)
+            Analyze.new(action_data, increments, @mem ?  "#{controller} in kilobytes" : "objects by action in /#{controller}_controller").draw          
+          
+            # in each action, by object class
+            action_data.keys.each do |action|
+              action = "unknown" if action.to_s == ""
+              puts "    class for #{action} action"
+              Dir.descend(action) do
+                class_data, increments = aggregate(data, /^#{controller}#{"\/#{action}" unless action == "unknown"}($|\/|::::)/, 
+                  /::::(.*)/)
+                Analyze.new(class_data, increments, "objects by class in /#{controller}/#{action}").draw
+              end
+            end unless @mem
+          
+          end          
+        end
+      end    
+    end 
+    
+  end
+end