sheap 0.1.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0f61e05e0d4dbe85f032a53a7b61a9d5446d1f7bddd599ed47b46c132c3615ca
-  data.tar.gz: f2c2aa947a8c2110df25add16f97b27a4a0df10473dd4e73c98a4634899489b1
+  metadata.gz: 84cd499accc73ed5e94564c3703fd2ed19dfb380b1c62cbccb35a207ab670e1e
+  data.tar.gz: b44e356f1d76c36b1cd4bb335594765aafafcdf3e21af96b7cecda3bb3617fdb
 SHA512:
-  metadata.gz: d75e2dce655e44a761c0f3a23c29afd59fe2e535f179b286175e1685d3e4552bf3c4a9eefc21495f1be67a31084db6ceb6a400f7fe40ac98bf21b0751665c50d
-  data.tar.gz: 5410d3da66d793c4c552878738726e15e34341c017f62bc8c992baebab2933fe3780a9efec522717c8c01ee02ac7bc9cdbbc65c7c9691d69861299b4fda26cb3
+  metadata.gz: 69fdf739e7088d998f3968024f2fe2b086ba79d2a4707f3dc924a87a079f40c2b89958647ef92fafa8f7a9f513377bae229c715178071a2c19c30384c81374ec
+  data.tar.gz: 6b29e443a217dab6b283c9026b85c1ac358fca2c8653926774cb7356eff8693e61652c1d4bad1032539c9cb0e37fba56b2dc3267d334a38c6105d2bcb8bde69f

data/README.md

@@ -1,24 +1,100 @@
 # Sheap
 
-TODO: Delete this and the text below, and describe your gem
+Sheap is a library for interactively exploring Ruby Heap dumps. Sheap contains a command-line tool and a library for use in IRB. 
 
-Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/sheap`. To experiment with that code, run `bin/console` for an interactive prompt.
+Some examples of things you can do with Sheap:
+- Find all retained objects between two heap dumps, and analyze them by their properties
+- Inspect individual objects in a heap dump, and interrogate the objects it references and the objects that reference it (reverse references)
+- For a given object, discover all paths back to the root of the heap, which can help you understand why an object is retained.
 
-## Installation
-
-TODO: Replace `UPDATE_WITH_YOUR_GEM_NAME_PRIOR_TO_RELEASE_TO_RUBYGEMS_ORG` with your gem name right after releasing it to RubyGems.org. Please do not do it earlier due to security reasons. Alternatively, replace this section with instructions to install your gem from git if you don't plan to release to RubyGems.org.
-
-Install the gem and add to the application's Gemfile by executing:
+Why Ruby heap dumps, briefly:
+- Ruby heap dumps are a snapshot of the state of the Ruby VM at a given point in time, which can be useful for understanding memory-related behavior such as bloat and retention issues.
+- The heap contains objects that may be familiar to your application (constants, classes, instances of classes, and primitives like strings and arrays), as well as objects that are internal to the Ruby VM, such as instruction sequences and call caches.
+- Ruby's garbage collector is a mark-and-sweep collector, which means that it starts at the root of the heap and marks all objects that are reachable from the root. It then sweeps the heap, freeing any objects that were not marked. This means that any object that is reachable from the root is retained, and any object that is not reachable from the root is freed. This is why it's useful to find all objects that are retained between two heap dumps (and thus multiple GC runs), inspect their properties, and understand their paths back to the root of the heap.
 
-    $ bundle add UPDATE_WITH_YOUR_GEM_NAME_PRIOR_TO_RELEASE_TO_RUBYGEMS_ORG
-
-If bundler is not being used to manage dependencies, install the gem by executing:
+## Installation
 
-    $ gem install UPDATE_WITH_YOUR_GEM_NAME_PRIOR_TO_RELEASE_TO_RUBYGEMS_ORG
+You can `gem install sheap` to get sheap as a library and command line tool. You can also download `lib/sheap.rb` to a remote server and require it as a standalone file from IRB.
 
 ## Usage
 
-TODO: Write usage instructions here
+Using the command line will open an IRB session with the heap loaded. You can then use the `$diff`, `$before`, and `$after` variable to explore the heap.
+
+```console
+$ sheap [HEAP_BEFORE.dump] [HEAP_AFTER.dump]
+```
+
+To use directly with IRB:
+
+```ruby
+# $ irb
+
+require './lib/sheap'
+
+# Create a diff of two heap dumps
+$diff = Sheap::Diff.new('tmp/heap_before.dump', 'tmp/heap_after.dump')
+
+# Find all retained objects and count by type
+$diff.retained.map(&:type_str).tally.sort_by(&:last)
+# => [["DATA", 1], ["FILE", 1], ["IMEMO", 4], ["STRING", 4], ["ARRAY", 10000]]
+
+# Find the 4 largest arrays in the 'after' heap dump
+>> $diff.after.arrays.sort_by(&:length).last(5)
+# =>
+# [#<ARRAY 0x100ec0440  (512 refs)>,
+#  #<ARRAY 0x100ec9270  (512 refs)>,
+#  #<ARRAY 0x100f4b450  (512 refs)>,
+#  #<ARRAY 0x11bc6d5b0  (512 refs)>,
+#  #<ARRAY 0x11c137960  (10000 refs)>]
+
+# Grab and examine just the largest array
+large_arr = $diff.after.arrays.max_by(&:length)
+# =>
+# #<ARRAY 0x1023effc8
+#  type="ARRAY",
+#  shape_id=0,
+#  slot_size=40,
+#  class=#<CLASS 0x100e43350 Array (252 refs)>,
+#  length=10000,
+#  references=(10000 refs),
+#  memsize=89712,
+#  flags=wb_protected>
+
+# Is it old?
+large_arr.old?
+# => false
+
+# Find the first of its references
+large_arr.references.first
+# =>
+# #<ARRAY 0x11c13fdb8
+#  type="ARRAY",
+#  shape_id=0,
+#  slot_size=40,
+#  class=#<CLASS 0x100e43350 Array (252 refs)>,
+#  length=0,
+#  embedded=true,
+#  memsize=40,
+#  flags=wb_protected>
+
+# Reference that same object by address
+$diff.after.at("0x11c13fdb8")
+# =>
+# #<ARRAY 0x11c13fdb8
+#  type="ARRAY",
+#  ...
+
+# Show that object's path back to the root of the heap
+$diff.after.find_path($diff.after.at("0x11c13fdb8"))
+# => [#<ROOT global_tbl (13 refs)>, #<ARRAY 0x1023effc8 (10000 refs)>, #<ARRAY 0x11c13fdb8>]
+```
+
+### Generating heap dumps
+
+Sheap on its own will not generate heap dumps for you. Some options for generating heap dumps:
+
+- `ObjectSpace.dump_all(output: open("tmp/snapshot1.dump", "w"))`
+- [Derailed Benchmarks](https://github.com/zombocom/derailed_benchmarks) `bundle exec derailed exec perf:heap_diff` produces 3 generations of heap dumps.
 
 ## Development
 

data/lib/sheap/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 class Sheap
-  VERSION = "0.1.0"
+  VERSION = "0.3.0"
 end

data/lib/sheap.rb

@@ -15,38 +15,107 @@ class Sheap
     FileUtils.mkdir_p(@dir)
   end
 
+  module Collection
+    def class_named(name)
+      filter do |obj|
+        obj.json.include?(name) &&
+          obj.type_str == "CLASS" &&
+          obj.name == name
+      end
+    end
+
+    def instances_of(klass)
+      addr = klass.address
+      filter do |obj|
+        obj.json.include?(addr) &&
+          obj.class_addr == addr
+      end
+    end
+
+    def of_type(type)
+      type = type.to_s.upcase
+      filter { |o| o.json.include?(type) && o.type_str == type }
+    end
+
+    def of_imemo_type(type)
+      type = type.to_s.downcase
+      filter { |o| o.json.include?(type) && o.imemo_type == type }
+    end
+
+    def classes; of_type("CLASS"); end
+    def icasses; of_type("ICLASS"); end
+    def modules; of_type("MODULE"); end
+    def imemos; of_type("IMEMO"); end
+
+    def strings; of_type("STRING"); end
+    def hashes; of_type("HASH"); end
+    def arrays; of_type("ARRAY"); end
+
+    def plain_objects; of_type("OBJECT"); end
+    def structs; of_type("STRUCT"); end
+    def datas; of_type("DATA"); end
+    def files; of_type("FILE"); end
+
+    def regexps; of_type("REGEXP"); end
+    def matches; of_type("MATCH"); end
+
+    def bignums; of_type("BIGNUM"); end
+    def symbols; of_type("SYMBOL"); end
+    def floats; of_type("FLOAT"); end
+    def rationals; of_type("RATIONAL"); end
+    def complexes; of_type("COMPLEX"); end
+
+    # imemo types
+    def iseqs; of_imemo_type("iseq"); end
+    def callcaches; of_imemo_type("callcache"); end
+    def constcaches; of_imemo_type("constcache"); end
+    def callinfos; of_imemo_type("callinfo"); end
+    def crefs; of_imemo_type("cref"); end
+    def ments; of_imemo_type("ment"); end
+  end
+
   class Diff
-    attr_reader :before, :after
-    def initialize(before, after)
-      @before = before
-      @after = after
+    include Collection
+
+    attr_reader :before, :after, :later
+    def initialize(before, after, later = nil)
+      @before = Heap.wrap(before)
+      @after = Heap.wrap(after)
+      @later = Heap.wrap(later) if later
     end
 
     def retained
-      @retained ||= after.objects - before.objects
+      @retained ||= HeapObjectCollection.new(calculate_retained, @after)
     end
-  end
+    alias objects retained
 
-  def self.load
-    Dir["sheap/*"].map do |file|
-      Heap.new(file)
+    def filter(&block)
+      retained.filter(&block)
     end
-  end
 
-  def self.load_diff
-    before = Heap.new("sheap/snapshot-0.dump")
-    after = Heap.new("sheap/snapshot-1.dump")
-    Diff.new(before, after)
-  end
+    def inspect
+      "#<#{self.class} (#{objects.size} objects)>"
+    end
 
-  def snapshot(gc: true)
-    3.times { GC.start } if gc
+    private
 
-    output = File.join(@dir, "snapshot-#{@idx}.dump")
-    File.open(output, "w") do |file|
-      ObjectSpace.dump_all(output: file)
+    def calculate_retained
+      set = Set.new
+      @after.objects.each do |obj|
+        set.add(obj)
+      end
+      @before.objects.each do |obj|
+        set.delete(obj)
+      end
+      if @later
+        later_set = Set.new(@later.objects)
+        set.select! do |obj|
+          later_set.include?(obj)
+        end
+      end
+
+      set.to_a
     end
-    @idx += 1
   end
 
   class << self
@@ -61,6 +130,82 @@ class Sheap
 
   EMPTY_ARRAY = [].freeze
 
+  class HeapObjectCollection
+    include Enumerable
+    include Collection
+
+    attr_reader :heap, :objects
+
+    def initialize(objects, heap = nil)
+      objects = objects.to_a unless objects.instance_of?(Array)
+      @objects = objects
+      @heap = heap || objects.first&.heap
+    end
+
+    def filter(&block)
+      HeapObjectCollection.new(@objects.select(&block), @heap)
+    end
+    alias select filter
+
+    def sample(n = nil)
+      if n
+        HeapObjectCollection.new(@objects.sample(n))
+      else
+        @objects.sample
+      end
+    end
+
+    def [](*args)
+      @objects[*args]
+    end
+
+    def last(n = nil)
+      if n
+        HeapObjectCollection.new(@objects.last(n))
+      else
+        objects.last
+      end
+    end
+
+    def each(&block)
+      @objects.each(&block)
+    end
+
+    def length
+      @objects.length
+    end
+    alias size length
+    def count(&block)
+      @objects.count(&block)
+    end
+
+    def pretty_print(q)
+      q.group(1, '[', ']') {
+        if size <= 20
+          q.seplist(self) {|v|
+            q.pp v
+          }
+        else
+          preview = 4
+          q.seplist(first(preview)) {|v|
+            q.pp v
+          }
+          q.comma_breakable
+          q.text "... (#{size - preview} more)"
+        end
+      }
+    end
+
+    def inspect
+      "#<#{self.class} (#{size} objects)>"
+    end
+
+    def to_a
+      @objects
+    end
+    alias to_ary to_a
+  end
+
   class HeapObject
     attr_reader :heap, :json
 
@@ -73,6 +218,10 @@ class Sheap
       @json[/"type":"([A-Z]+)"/, 1]
     end
 
+    def root?
+      @json.include?('"type":"ROOT"')
+    end
+
     def address
       @json[/"address":"(0x[0-9a-f]+)"/, 1] || @json[/"root":"([a-z_]+)"/, 1]
     end
@@ -87,13 +236,19 @@ class Sheap
     end
 
     def references
-      referenced_addrs.map do |addr|
-        @heap.at(addr)
-      end
+      HeapObjectCollection.new(
+        referenced_addrs.map do |addr|
+          @heap.at(addr)
+        end,
+        heap
+      )
     end
 
     def inverse_references
-      @heap.inverse_references[address] || EMPTY_ARRAY
+      HeapObjectCollection.new(
+        (@heap.inverse_references[address] || EMPTY_ARRAY),
+        heap
+      )
     end
 
     def data
@@ -109,17 +264,21 @@ class Sheap
     end
 
     def imemo_type
-      @json[/"imemo_type":"([a-z]+)"/, 1]
+      @json[/"imemo_type":"([a-z_]+)"/, 1]
     end
 
     def struct
-      @json[/"struct":"([a-zA-Z]+)"/, 1]
+      @json[/"struct":"([^"]+)"/, 1]
     end
 
     def wb_protected?
       @json.include?('"wb_protected":true')
     end
 
+    def old?
+      @json.include?('"old":true')
+    end
+
     def name
       data["name"]
     end
@@ -133,23 +292,30 @@ class Sheap
       heap.instances_of(self)
     end
 
+    def superclass
+      heap.at(data["superclass"])
+    end
+
     def inspect
       type_str = self.type_str
-      s = +"<#{type_str} #{address}"
+      s = +"<#{type_str} #{address} #{inspect_hint}>"
+    end
 
+    def inspect_hint
+      s = +""
       case type_str
       when "CLASS"
-        s << " " << (name || "(anonymous)")
+        s << (name || "(anonymous)")
       when "MODULE"
-        s << " " << (name || "(anonymous)")
+        s << (name || "(anonymous)")
       when "STRING"
-        s << " " << data["value"].inspect
+        s << data["value"].inspect
       when "IMEMO"
-        s << " " << (imemo_type || "unknown")
+        s << (imemo_type || "unknown")
       when "OBJECT"
-        s << " " << (klass.name || "(#{klass.address})")
+        s << (klass.name || "(#{klass.address})")
       when "DATA"
-        s << " " << struct.to_s
+        s << struct.to_s
       end
 
       refs = referenced_addrs
@@ -157,7 +323,42 @@ class Sheap
         s << " (#{referenced_addrs.size} refs)"
       end
 
-      s << ">"
+      s
+    end
+
+    def pretty_print(q)
+      current_depth = q.current_group.depth
+      q.group(1, "#<#{type_str}", '>') do
+        q.text " "
+        q.text address
+        if current_depth <= 1
+          data = self.data
+          attributes = data.keys - ["address"]
+          q.seplist(attributes, lambda { q.text ',' }) {|v|
+            q.breakable
+            q.text v
+            q.text "="
+            q.group(1) {
+              q.breakable ''
+              case v
+              when "class"
+                q.pp klass
+              when "superclass"
+                q.pp superclass
+              when "flags"
+                q.text flags.keys.join("|")
+              when "references"
+                q.text "(#{referenced_addrs.size} refs)"
+              else
+                q.pp data[v]
+              end
+            }
+          }
+        else
+          q.breakable
+          q.text inspect_hint
+        end
+      end
     end
 
     def value
@@ -198,9 +399,27 @@ class Sheap
     def hash
       address.hash
     end
+
+    def method_missing(name, *args)
+      if value = data[name.to_s]
+        value
+      else
+        super
+      end
+    end
+
+    def respond_to_missing?(name, *)
+      data.key?(name.to_s) || super
+    end
+
+    def [](key)
+      data[key.to_s]
+    end
   end
 
   class Heap
+    include Collection
+
     attr_reader :filename
 
     def initialize(filename)
@@ -210,15 +429,29 @@ class Sheap
     def each_object
       return enum_for(__method__) unless block_given?
 
-      File.open(filename) do |file|
+      open_file do |file|
         file.each_line do |json|
           yield HeapObject.new(self, json)
         end
       end
     end
 
+    def open_file(&block)
+      # FIXME: look for magic header
+      if filename.end_with?(".gz")
+        require "zlib"
+        Zlib::GzipReader.open(filename, &block)
+      else
+        File.open(filename, &block)
+      end
+    end
+
     def objects
-      @objects ||= each_object.to_a
+      @objects ||= HeapObjectCollection.new(each_object.to_a, self)
+    end
+
+    def filter(&block)
+      objects.filter(&block)
     end
 
     def objects_by_addr
@@ -247,33 +480,54 @@ class Sheap
         end
     end
 
-    def at(addr)
-      objects_by_addr[addr]
+    def roots
+      of_type("ROOT")
     end
 
-    def class_named(name)
-      objects.select do |obj|
-        obj.json.include?(name) &&
-          obj.type_str == "CLASS" &&
-          obj.name == name
+    # finds a path from `start_address` through the inverse_references hash
+    # and so the end_address will be the object that's closer to the root
+    def find_path(start_addresses, end_addresses = nil)
+      if end_addresses.nil?
+        end_addresses = start_addresses
+        start_addresses = roots
       end
-    end
+      start_addresses = Array(start_addresses)
+      end_addresses = Array(end_addresses)
 
-    def instances_of(klass)
-      addr = klass.address
-      objects.select do |obj|
-        obj.json.include?(addr) &&
-          obj.class_addr == addr
+      q = start_addresses.map{|x| [x] }
+
+      visited = Set.new
+      while !q.empty?
+        current_path = q.shift
+        current_address = current_path.last
+
+        if end_addresses.include?(current_address)
+          return current_path.map{|addr| addr}
+        end
+
+        if !visited.include?(current_address)
+          visited.add(current_address)
+
+          current_references = current_address.references
+
+          current_references.each do |obj|
+            q.push([*current_path, obj])
+          end
+        end
       end
+      nil
     end
 
-    def of_type(type)
-      type = type.to_s.upcase
-      objects.select { |o| o.type_str == type }
+    def at(addr)
+      objects_by_addr[addr]
     end
 
     def inspect
       "#<#{self.class} (#{objects.size} objects)>"
     end
+
+    def self.wrap(heap)
+      self === heap ? heap : new(heap)
+    end
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: sheap
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.3.0
 platform: ruby
 authors:
 - John Hawthorn
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2023-10-13 00:00:00.000000000 Z
+date: 2024-05-26 00:00:00.000000000 Z
 dependencies: []
 description: A set of helpers for analyzing the output of ObjectSpace.dump_all
 email:
@@ -26,6 +26,7 @@ files:
 - exe/sheap
 - lib/sheap.rb
 - lib/sheap/version.rb
+- tmp/.keep
 homepage: https://github.com/jhawthorn/sheap
 licenses:
 - MIT
@@ -48,7 +49,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.4.10
+rubygems_version: 3.5.3
 signing_key:
 specification_version: 4
 summary: A helpers for heap dumps