sheap 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0f61e05e0d4dbe85f032a53a7b61a9d5446d1f7bddd599ed47b46c132c3615ca
-  data.tar.gz: f2c2aa947a8c2110df25add16f97b27a4a0df10473dd4e73c98a4634899489b1
+  metadata.gz: 63584faef266eb7722518ee68bbb9cb757786a6aec38779889466b3d215eb542
+  data.tar.gz: f4979afbb9ce9f1f7ae8aabb58a8b69e7fc86b676c35a451774311a8d6d212d1
 SHA512:
-  metadata.gz: d75e2dce655e44a761c0f3a23c29afd59fe2e535f179b286175e1685d3e4552bf3c4a9eefc21495f1be67a31084db6ceb6a400f7fe40ac98bf21b0751665c50d
-  data.tar.gz: 5410d3da66d793c4c552878738726e15e34341c017f62bc8c992baebab2933fe3780a9efec522717c8c01ee02ac7bc9cdbbc65c7c9691d69861299b4fda26cb3
+  metadata.gz: 5388e77ef86d256762f7388836f6aa2f34837e0a87da9e9fa842156fca972c475b87c506c7310cbde3739343d250bcc11714c875785627fd557da7bd0a7606e7
+  data.tar.gz: b3696454a64657a42c07c331d659938933417fb611ef5494127d76c9aa4b83ee2ef057c0c0e2d220d8cf27a1b822c9e5300b0a1fee9c1706ab07283335e21175

data/README.md

@@ -1,24 +1,88 @@
 # Sheap
 
-TODO: Delete this and the text below, and describe your gem
+Sheap is a library for interactively exploring Ruby Heap dumps. Sheap contains contains a command-line tool and a library for use in IRB. 
 
-Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/sheap`. To experiment with that code, run `bin/console` for an interactive prompt.
+Some examples of things you can do with Sheap:
+- Find all retained objects between two heap dumps, and analyze them by their properties
+- Inspect individual objects in a heap dump, and interrogate the objects it references and the objects that reference it (reverse references)
+- For a given object, discover all paths back to the root of the heap, which can help you understand why an object is retained.
 
-## Installation
-
-TODO: Replace `UPDATE_WITH_YOUR_GEM_NAME_PRIOR_TO_RELEASE_TO_RUBYGEMS_ORG` with your gem name right after releasing it to RubyGems.org. Please do not do it earlier due to security reasons. Alternatively, replace this section with instructions to install your gem from git if you don't plan to release to RubyGems.org.
-
-Install the gem and add to the application's Gemfile by executing:
+Why Ruby heap dumps, briefly:
+- Ruby heap dumps are a snapshot of the state of the Ruby VM at a given point in time, which can be useful for understanding memory-related behavior such as bloat and retention issues.
+- The heap contains objects that may be familiar to your application (constants, classes, instances of classes, and primitives like strings and arrays), as well as objects that are internal to the Ruby VM, such as instruction sequences and call caches.
+- Ruby's garbage collector is a mark-and-sweep collector, which means that it starts at the root of the heap and marks all objects that are reachable from the root. It then sweeps the heap, freeing any objects that were not marked. This means that any object that is reachable from the root is retained, and any object that is not reachable from the root is freed. This is why it's useful to find all objects that are retained between two heap dumps (and thus multiple GC runs), inspect their properties, and understand their paths back to the root of the heap.
 
-    $ bundle add UPDATE_WITH_YOUR_GEM_NAME_PRIOR_TO_RELEASE_TO_RUBYGEMS_ORG
-
-If bundler is not being used to manage dependencies, install the gem by executing:
+## Installation
 
-    $ gem install UPDATE_WITH_YOUR_GEM_NAME_PRIOR_TO_RELEASE_TO_RUBYGEMS_ORG
+You can `gem install sheap` to get sheap as a library and command line tool. You can also download `lib/sheap.rb` to a remote server and require it as a standalone file from IRB.
 
 ## Usage
 
-TODO: Write usage instructions here
+Using the command line will open an IRB session with the heap loaded. You can then use the `$diff`, `$before`, and `$after` variable to explore the heap.
+
+```console
+$ sheap [HEAP_BEFORE.dump] [HEAP_AFTER.dump]
+```
+
+To use directly with IRB:
+
+```ruby
+# $ irb
+
+require './lib/sheap'
+
+# Create a diff of two heap dumps
+$diff = Sheap::Diff.new('tmp/heap_before.dump', 'tmp/heap_after.dump')
+
+# Find all retained objects and count by type
+$diff.retained.map(&:type_str).tally.sort_by(&:last)
+# => [["DATA", 1], ["FILE", 1], ["IMEMO", 4], ["STRING", 4], ["ARRAY", 10000]]
+
+# Find the largest array in the 'after' heap dump
+$diff.after.of_type("ARRAY").sort_by { |o| o.data["length"] }.last
+# => <ARRAY 0x1023effc8 (10000 refs)>
+
+# Is it old?
+$diff.after.of_type("ARRAY").sort_by { |o| o.data["length"] }.last.old?
+# => false
+
+# What else can be learned about it
+$diff.after.of_type("ARRAY").sort_by { |o| o.data["length"] }.last.data
+# => 
+# { "address"=>"0x1023effc8",
+#   "type"=>"ARRAY",
+#   "shape_id"=>0,
+#   "slot_size"=>40,
+#   "class"=>"0x1024c33f0",
+#   "length"=>10000,
+#   "memsize"=>89712,
+#   "flags"=>{"wb_protected"=>true}
+#   "references" => [
+#      <ARRAY 0x1023efc80>, 
+#      <ARRAY 0x1023efc58>,
+#      <ARRAY 0x1023efc30>,
+#      <ARRAY 0x1023efc08>,
+#      <ARRAY 0x1023efbe0>,
+#      <ARRAY 0x1023efbb8>,
+#      # ...
+#   ]
+# }
+
+# Find the first of its references by address
+$diff.after.at("0x1023efc80")
+=> <ARRAY 0x1023efc80>
+
+# Show that object's path back to the root of the heap 
+$diff.after.find_path($diff.after.at("0x1023efc80"))
+# => [<ROOT global_tbl (13 refs)>, <ARRAY 0x1023effc8 (10000 refs)>, <ARRAY 0x1023efc80>]
+```
+
+### Generating heap dumps
+
+Sheap on its own will not generate heap dumps for you. Some options for generating heap dumps:
+
+- `ObjectSpace.dump_all(output: open("tmp/snapshot1.dump", "w"))`
+- [Derailed Benchmarks](https://github.com/zombocom/derailed_benchmarks) `bundle exec derailed exec perf:heap_diff` produces 3 generations of heap dumps.
 
 ## Development
 

data/lib/sheap/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 class Sheap
-  VERSION = "0.1.0"
+  VERSION = "0.2.0"
 end

data/lib/sheap.rb

@@ -15,38 +15,94 @@ class Sheap
     FileUtils.mkdir_p(@dir)
   end
 
+  module Collection
+    def class_named(name)
+      filter do |obj|
+        obj.json.include?(name) &&
+          obj.type_str == "CLASS" &&
+          obj.name == name
+      end
+    end
+
+    def instances_of(klass)
+      addr = klass.address
+      filter do |obj|
+        obj.json.include?(addr) &&
+          obj.class_addr == addr
+      end
+    end
+
+    def of_type(type)
+      type = type.to_s.upcase
+      filter { |o| o.json.include?(type) && o.type_str == type }
+    end
+
+    def classes; of_type("CLASS"); end
+    def icasses; of_type("ICLASS"); end
+    def modules; of_type("MODULE"); end
+    def imemos; of_type("IMEMO"); end
+
+    def strings; of_type("STRING"); end
+    def hashes; of_type("HASH"); end
+    def arrays; of_type("ARRAY"); end
+
+    def plain_objects; of_type("OBJECT"); end
+    def structs; of_type("STRUCT"); end
+    def datas; of_type("DATA"); end
+    def files; of_type("FILE"); end
+
+    def regexps; of_type("REGEXP"); end
+    def matches; of_type("MATCH"); end
+
+    def bignums; of_type("BIGNUM"); end
+    def symbols; of_type("SYMBOL"); end
+    def floats; of_type("FLOAT"); end
+    def rationals; of_type("RATIONAL"); end
+    def complexes; of_type("COMPLEX"); end
+  end
+
   class Diff
-    attr_reader :before, :after
-    def initialize(before, after)
-      @before = before
-      @after = after
+    include Collection
+
+    attr_reader :before, :after, :later
+    def initialize(before, after, later = nil)
+      @before = Heap.wrap(before)
+      @after = Heap.wrap(after)
+      @later = Heap.wrap(later) if later
     end
 
     def retained
-      @retained ||= after.objects - before.objects
+      @retained ||= HeapObjectCollection.new(calculate_retained, @after)
     end
-  end
+    alias objects retained
 
-  def self.load
-    Dir["sheap/*"].map do |file|
-      Heap.new(file)
+    def filter(&block)
+      retained.filter(&block)
     end
-  end
 
-  def self.load_diff
-    before = Heap.new("sheap/snapshot-0.dump")
-    after = Heap.new("sheap/snapshot-1.dump")
-    Diff.new(before, after)
-  end
+    def inspect
+      "#<#{self.class} (#{objects.size} objects)>"
+    end
 
-  def snapshot(gc: true)
-    3.times { GC.start } if gc
+    private
+
+    def calculate_retained
+      set = Set.new
+      @after.objects.each do |obj|
+        set.add(obj)
+      end
+      @before.objects.each do |obj|
+        set.delete(obj)
+      end
+      if @later
+        later_set = Set.new(@later.objects)
+        set.select! do |obj|
+          later_set.include?(obj)
+        end
+      end
 
-    output = File.join(@dir, "snapshot-#{@idx}.dump")
-    File.open(output, "w") do |file|
-      ObjectSpace.dump_all(output: file)
+      set.to_a
     end
-    @idx += 1
   end
 
   class << self
@@ -61,6 +117,76 @@ class Sheap
 
   EMPTY_ARRAY = [].freeze
 
+  class HeapObjectCollection
+    include Enumerable
+    include Collection
+
+    attr_reader :heap, :objects
+
+    def initialize(objects, heap = nil)
+      objects = objects.to_a unless objects.instance_of?(Array)
+      @objects = objects
+      @heap = heap || objects.first&.heap
+    end
+
+    def filter(&block)
+      HeapObjectCollection.new(@objects.select(&block), @heap)
+    end
+    alias select filter
+
+    def sample(n = nil)
+      if n
+        HeapObjectCollection.new(@objects.sample(n))
+      else
+        @objects.sample
+      end
+    end
+
+    def last(n = nil)
+      if n
+        HeapObjectCollection.new(@objects.last(n))
+      else
+        objects.last
+      end
+    end
+
+    def each(&block)
+      @objects.each(&block)
+    end
+
+    def length
+      @objects.length
+    end
+    alias size length
+    alias count length
+
+    def pretty_print(q)
+      q.group(1, '[', ']') {
+        if size <= 20
+          q.seplist(self) {|v|
+            q.pp v
+          }
+        else
+          preview = 4
+          q.seplist(first(preview)) {|v|
+            q.pp v
+          }
+          q.comma_breakable
+          q.text "... (#{size - preview} more)"
+        end
+      }
+    end
+
+    def inspect
+      "#<#{self.class} (#{size} objects)>"
+    end
+
+    def to_a
+      @objects
+    end
+    alias to_ary to_a
+  end
+
   class HeapObject
     attr_reader :heap, :json
 
@@ -73,6 +199,10 @@ class Sheap
       @json[/"type":"([A-Z]+)"/, 1]
     end
 
+    def root?
+      @json.include?('"type":"ROOT"')
+    end
+
     def address
       @json[/"address":"(0x[0-9a-f]+)"/, 1] || @json[/"root":"([a-z_]+)"/, 1]
     end
@@ -87,13 +217,19 @@ class Sheap
     end
 
     def references
-      referenced_addrs.map do |addr|
-        @heap.at(addr)
-      end
+      HeapObjectCollection.new(
+        referenced_addrs.map do |addr|
+          @heap.at(addr)
+        end,
+        heap
+      )
     end
 
     def inverse_references
-      @heap.inverse_references[address] || EMPTY_ARRAY
+      HeapObjectCollection.new(
+        (@heap.inverse_references[address] || EMPTY_ARRAY),
+        heap
+      )
     end
 
     def data
@@ -109,17 +245,21 @@ class Sheap
     end
 
     def imemo_type
-      @json[/"imemo_type":"([a-z]+)"/, 1]
+      @json[/"imemo_type":"([a-z_]+)"/, 1]
     end
 
     def struct
-      @json[/"struct":"([a-zA-Z]+)"/, 1]
+      @json[/"struct":"([^"]+)"/, 1]
     end
 
     def wb_protected?
       @json.include?('"wb_protected":true')
     end
 
+    def old?
+      @json.include?('"old":true')
+    end
+
     def name
       data["name"]
     end
@@ -133,23 +273,30 @@ class Sheap
       heap.instances_of(self)
     end
 
+    def superclass
+      heap.at(data["superclass"])
+    end
+
     def inspect
       type_str = self.type_str
-      s = +"<#{type_str} #{address}"
+      s = +"<#{type_str} #{address} #{inspect_hint}>"
+    end
 
+    def inspect_hint
+      s = +""
       case type_str
       when "CLASS"
-        s << " " << (name || "(anonymous)")
+        s << (name || "(anonymous)")
       when "MODULE"
-        s << " " << (name || "(anonymous)")
+        s << (name || "(anonymous)")
       when "STRING"
-        s << " " << data["value"].inspect
+        s << data["value"].inspect
       when "IMEMO"
-        s << " " << (imemo_type || "unknown")
+        s << (imemo_type || "unknown")
       when "OBJECT"
-        s << " " << (klass.name || "(#{klass.address})")
+        s << (klass.name || "(#{klass.address})")
       when "DATA"
-        s << " " << struct.to_s
+        s << struct.to_s
       end
 
       refs = referenced_addrs
@@ -157,7 +304,42 @@ class Sheap
         s << " (#{referenced_addrs.size} refs)"
       end
 
-      s << ">"
+      s
+    end
+
+    def pretty_print(q)
+      current_depth = q.current_group.depth
+      q.group(1, "#<#{type_str}", '>') do
+        q.text " "
+        q.text address
+        if current_depth <= 1
+          data = self.data
+          attributes = data.keys - ["address"]
+          q.seplist(attributes, lambda { q.text ',' }) {|v|
+            q.breakable
+            q.text v
+            q.text "="
+            q.group(1) {
+              q.breakable ''
+              case v
+              when "class"
+                q.pp klass
+              when "superclass"
+                q.pp superclass
+              when "flags"
+                q.text flags.keys.join("|")
+              when "references"
+                q.text "(#{referenced_addrs.size} refs)"
+              else
+                q.pp data[v]
+              end
+            }
+          }
+        else
+          q.breakable
+          q.text inspect_hint
+        end
+      end
     end
 
     def value
@@ -198,9 +380,27 @@ class Sheap
     def hash
       address.hash
     end
+
+    def method_missing(name, *args)
+      if value = data[name.to_s]
+        value
+      else
+        super
+      end
+    end
+
+    def respond_to_missing?(name, *)
+      data.key?(name.to_s) || super
+    end
+
+    def [](key)
+      data[key.to_s]
+    end
   end
 
   class Heap
+    include Collection
+
     attr_reader :filename
 
     def initialize(filename)
@@ -210,15 +410,29 @@ class Sheap
     def each_object
       return enum_for(__method__) unless block_given?
 
-      File.open(filename) do |file|
+      open_file do |file|
         file.each_line do |json|
           yield HeapObject.new(self, json)
         end
       end
     end
 
+    def open_file(&block)
+      # FIXME: look for magic header
+      if filename.end_with?(".gz")
+        require "zlib"
+        Zlib::GzipReader.open(filename, &block)
+      else
+        File.open(filename, &block)
+      end
+    end
+
     def objects
-      @objects ||= each_object.to_a
+      @objects ||= HeapObjectCollection.new(each_object.to_a, self)
+    end
+
+    def filter(&block)
+      objects.filter(&block)
     end
 
     def objects_by_addr
@@ -247,33 +461,54 @@ class Sheap
         end
     end
 
-    def at(addr)
-      objects_by_addr[addr]
+    def roots
+      of_type("ROOT")
     end
 
-    def class_named(name)
-      objects.select do |obj|
-        obj.json.include?(name) &&
-          obj.type_str == "CLASS" &&
-          obj.name == name
+    # finds a path from `start_address` through the inverse_references hash
+    # and so the end_address will be the object that's closer to the root
+    def find_path(start_addresses, end_addresses = nil)
+      if end_addresses.nil?
+        end_addresses = start_addresses
+        start_addresses = roots
       end
-    end
+      start_addresses = Array(start_addresses)
+      end_addresses = Array(end_addresses)
 
-    def instances_of(klass)
-      addr = klass.address
-      objects.select do |obj|
-        obj.json.include?(addr) &&
-          obj.class_addr == addr
+      q = start_addresses.map{|x| [x] }
+
+      visited = Set.new
+      while !q.empty?
+        current_path = q.shift
+        current_address = current_path.last
+
+        if end_addresses.include?(current_address)
+          return current_path.map{|addr| addr}
+        end
+
+        if !visited.include?(current_address)
+          visited.add(current_address)
+
+          current_references = current_address.references
+
+          current_references.each do |obj|
+            q.push([*current_path, obj])
+          end
+        end
       end
+      nil
     end
 
-    def of_type(type)
-      type = type.to_s.upcase
-      objects.select { |o| o.type_str == type }
+    def at(addr)
+      objects_by_addr[addr]
     end
 
     def inspect
       "#<#{self.class} (#{objects.size} objects)>"
     end
+
+    def self.wrap(heap)
+      self === heap ? heap : new(heap)
+    end
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: sheap
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
 platform: ruby
 authors:
 - John Hawthorn
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2023-10-13 00:00:00.000000000 Z
+date: 2024-02-22 00:00:00.000000000 Z
 dependencies: []
 description: A set of helpers for analyzing the output of ObjectSpace.dump_all
 email:
@@ -26,6 +26,7 @@ files:
 - exe/sheap
 - lib/sheap.rb
 - lib/sheap/version.rb
+- tmp/.keep
 homepage: https://github.com/jhawthorn/sheap
 licenses:
 - MIT
@@ -48,7 +49,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.4.10
+rubygems_version: 3.5.3
 signing_key:
 specification_version: 4
 summary: A helpers for heap dumps