majo 0.0.3 → 0.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 30b12103910ff82b8073d3882debca86928bc05d03e318b8c640e0c3b6f4c6ee
-  data.tar.gz: 851e43d87d7ba4020b5eb04487ae9859522bba5bd7f7a6765d02432efe2b8d74
+  metadata.gz: 84aa655c3585c95f2e639690246b2946114208ce08b979d00d46e6a93efef60c
+  data.tar.gz: 0073efd2fd643f97219ca17be43259960cfb514ab864cbb6f0b0f9378d3f7a9f
 SHA512:
-  metadata.gz: ee86c8d4f321e54646ec82ee79168a8071ff641be47f9fd006ab70c9f254fbd958e6667c1e9dde36b00e911eee17d970a09a785dd53912f9bc4383101e649957
-  data.tar.gz: 3b707eee86c906409bd26f9b15e0e35aa1e6516e566b7d0f7e526b4fc76d74478110f75aa77b23041133d8b966999882c46d60515617121d7eec9cd41d9c3ac1
+  metadata.gz: 7d0cb81a092b6c0db17c5db53d2a2404fc554c1545bf8ab7b6546679ba7bd9a64e603af30e2fe7d1d5f90fe0848aebba490918d5101a179b6ac309bf21e3b878
+  data.tar.gz: 630052b3e6c3558c7ce2628a17528518e3770188fca26b26ff1f51a69c6d3d14e339b03814a93173ed134e6565d4e6f1aa1a07249961a70360c8903949943889

data/ARCHITECTURE.md

@@ -0,0 +1,25 @@
+# Architecture of Majo
+
+Majo uses internal TracePoint hooks, `RUBY_INTERNAL_EVENT_NEWOBJ` and `RUBY_INTERNAL_EVENT_FREEOBJ`, to trace object allocation and free events.
+We can use `ObjectSpace.trace_object_allocations_start` method in the Ruby-level, but this API is not suitable for Majo because it does not provide information about freed objects. Therefore, Majo uses these hooks in the C-level.
+
+## On allocation event
+
+When an object is allocated, Majo records the object into a st table with the path, line number, and so on.
+
+See `newobj_i` function in `majo.c`.
+
+## On free event
+
+When an object is freed, Majo moves the allocation information to an array from the st table if it's a long-lived object.
+It determines whether the object is long-lived by checking the GC count. If the object is not swept by a GC sweeping phase, it's a long-lived object.
+
+See `freeobj_i` function in `majo.c`.
+
+## Output format
+
+Majo provides three output formats: `color`, `monochrome`, and `csv`.
+
+`color` and `monochrome` are for human-readable output. They provide overview information about long-lived objects.
+
+`csv` is for machine-readable output. It provides the raw information. You can use this format to analyze the result with other tools.

data/README.md

@@ -1,24 +1,142 @@
-# Majo
+# Majo🧙‍♀️
 
-TODO: Delete this and the text below, and describe your gem
+A memory profiler focusing on long-lived objects.
 
-Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/majo`. To experiment with that code, run `bin/console` for an interactive prompt.
+## Motivation
 
-## Installation
+I created this gem because I wanted to reduce the maximum memory usage of a Ruby program.
+
+The existing memory profiler, `memory_profiler` gem, focuses on allocated and retained memory. This tool is useful to reduce allocations and retain memory, but it is not suitable for reducing the maximum memory usage.
 
-TODO: Replace `UPDATE_WITH_YOUR_GEM_NAME_IMMEDIATELY_AFTER_RELEASE_TO_RUBYGEMS_ORG` with your gem name right after releasing it to RubyGems.org. Please do not do it earlier due to security reasons. Alternatively, replace this section with instructions to install your gem from git if you don't plan to release to RubyGems.org.
+This gem solves this problem by focusing on long-lived objects. Long-lived objects are objects that are not garbage collected for a long time. These objects are the main cause of the maximum memory usage.
+
+## Installation
 
 Install the gem and add to the application's Gemfile by executing:
 
-    $ bundle add UPDATE_WITH_YOUR_GEM_NAME_IMMEDIATELY_AFTER_RELEASE_TO_RUBYGEMS_ORG
+    $ bundle add majo --group development
 
 If bundler is not being used to manage dependencies, install the gem by executing:
 
-    $ gem install UPDATE_WITH_YOUR_GEM_NAME_IMMEDIATELY_AFTER_RELEASE_TO_RUBYGEMS_ORG
+    $ gem install majo
 
 ## Usage
 
-TODO: Write usage instructions here
+Wrap the code you want to profile with `Majo.run`.
+Then, call `report` method to display the result.
+
+```ruby
+result = Majo.run do
+  code_to_profile
+end
+
+result.report
+```
+
+### Options
+
+You can pass the following options to `report` method.
+
+| Name        | Description              | Default                  | Type                                                                     |
+| ----------- | ------------------------ | ------------------------ | ------------------------------------------------------------------------ |
+| `out`       | Output file or IO        | `$stdout`                | `IO`, `String` or an object having `to_path` method (such as `Pathname`) |
+| `formatter` | The format of the result | `:color` or `monochrome` | `:color`, `:monochrome`, or `:csv`                                       |
+
+For example:
+
+```ruby
+result = Majo.run do
+  code_to_profile
+end
+
+result.report(out: "majo-result.csv", formatter: :csv)
+```
+
+### Result Example
+
+The result contains only long-lived objects, which are collected by the major GC.
+
+The example is as follows:
+
+```
+Total 15055372 bytes (159976 objects)
+
+Memory by file
+-----------------------------------
+10431760  /path/to/gems/rbs-3.5.1/lib/rbs/parser_aux.rb
+ 1966348  /path/to/gems/rbs-3.5.1/lib/rbs/environment_loader.rb
+  980344  /path/to/gems/rbs-3.5.1/lib/rbs/types.rb
+(snip)
+
+Memory by location
+-----------------------------------
+10431760  /path/to/gems/rbs-3.5.1/lib/rbs/parser_aux.rb:20
+ 1942812  /path/to/gems/rbs-3.5.1/lib/rbs/environment_loader.rb:159
+  249920  /path/to/gems/rbs-3.5.1/lib/rbs/types.rb:994
+(snip)
+
+Memory by class
+-----------------------------------
+4274028  String
+3556632  RBS::Location
+2197840  Array
+(snip)
+
+Objects by file
+-----------------------------------
+109126  /path/to/gems/rbs-3.5.1/lib/rbs/parser_aux.rb
+ 20813  /path/to/gems/rbs-3.5.1/lib/rbs/types.rb
+ 12236  /path/to/gems/rbs-3.5.1/lib/rbs/environment.rb
+(snip)
+
+Objects by location
+-----------------------------------
+109126  /path/to/gems/rbs-3.5.1/lib/rbs/parser_aux.rb:20
+  4458  /path/to/gems/rbs-3.5.1/lib/rbs/namespace.rb:24
+  4132  /path/to/gems/rbs-3.5.1/lib/rbs/types.rb:374
+(snip)
+
+Objects by class
+-----------------------------------
+52495  Array
+22435  RBS::Location
+11144  RBS::TypeName
+(snip)
+```
+
+The CSV format is as follows:
+
+```csv
+Object class path,Class path,Method ID,Path,Line,Alloc generation,Free generation,Memsize,Count
+Hash,,_parse_signature,/path/to/gems/rbs-3.5.1/lib/rbs/parser_aux.rb,20,20,22,160,3
+Hash,RBS::EnvironmentLoader,each_signature,/path/to/gems/rbs-3.5.1/lib/rbs/environment_loader.rb,159,20,22,160,1
+Hash,,_parse_signature,/path/to/gems/rbs-3.5.1/lib/rbs/parser_aux.rb,20,21,23,160,1
+(snip)
+```
+
+You can find the raw data in the CSV format. It is useful for further analysis. For example: [A spreadsheet for Majo result](https://docs.google.com/spreadsheets/d/1Qe6ZSJ58bNfLbA_eSuL9FJy89taNPt325qAJnLDorOE/edit?gid=533761210#gid=533761210)
+
+The columns are as follows:
+
+| Column name         | Description                                                        |
+| ------------------- | ------------------------------------------------------------------ |
+| `Object class path` | The class name of the allocated object                             |
+| `Class path`        | The class name of the receiver of the method allocating the object |
+| `Method ID`         | The method name allocating the object                              |
+| `Path`              | The file path of the method allocating the object                  |
+| `Line`              | The line number of the method allocating the object                |
+| `Alloc generation`  | The GC generation number when the object is allocated              |
+| `Free generation`   | The GC generation number when the object is freed                  |
+| `Memsize`           | The memory size of the object in bytes                             |
+| `Count`             | Number of objects allocated with the same conditions               |
+
+## Name
+
+The name "Majo" is a Japanese word `魔女` that means "witch". It is a wordplay on the word "major" in "major GC".
+
+## Thanks
+
+This gem is inspired by [memory_profiler](https://github.com/SamSaffron/memory_profiler) gem.
 
 ## Development
 

data/ext/majo/allocation_info.c

@@ -2,13 +2,12 @@
 
 static void majo_allocation_info_mark(void *ptr)
 {
-  // TODO
   majo_allocation_info *info = (majo_allocation_info*)ptr;
+  rb_gc_mark(info->result);
 }
 
 static void majo_allocation_info_free(majo_allocation_info *info) {
-  // TODO
-  ruby_xfree(info);
+  free(info);
 }
 
 static size_t majo_allocation_info_memsize(const void *ptr) {

data/ext/majo/allocation_info.h

@@ -11,6 +11,8 @@ typedef struct {
   size_t alloc_generation;
   size_t free_generation;
   size_t memsize;
+
+  VALUE result;
 } majo_allocation_info;
 
 VALUE

data/ext/majo/majo.c

@@ -39,7 +39,8 @@ internal_object_p(VALUE obj)
 static void
 newobj_i(VALUE tpval, void *data)
 {
-  majo_result *arg = (majo_result *)data;
+  VALUE res = (VALUE)data;
+  majo_result *arg = majo_check_result(res);
 
   rb_trace_arg_t *tparg = rb_tracearg_from_tracepoint(tpval);
   VALUE obj = rb_tracearg_object(tparg);
@@ -53,7 +54,6 @@ newobj_i(VALUE tpval, void *data)
   VALUE mid = rb_tracearg_method_id(tparg);
   VALUE klass = rb_tracearg_defined_class(tparg);
 
-  // TODO: when the st already has an entry for the value
   majo_allocation_info *info = (majo_allocation_info *)malloc(sizeof(majo_allocation_info));
 
   const char *path_cstr = RTEST(path) ? majo_make_unique_str(arg->str_table, RSTRING_PTR(path), RSTRING_LEN(path)) : 0;
@@ -66,6 +66,7 @@ newobj_i(VALUE tpval, void *data)
   const char *obj_class_path_cstr = RTEST(obj_class_path) ? majo_make_unique_str(arg->str_table, RSTRING_PTR(obj_class_path), RSTRING_LEN(obj_class_path)) : 0;
 
 
+  info->result = res;
   info->path = path_cstr;
   info->line = NUM2INT(line);
   info->mid = mid;
@@ -84,11 +85,9 @@ freeobj_i(VALUE tpval, void *data)
   st_data_t obj = (st_data_t)rb_tracearg_object(tparg);
   st_data_t v;
 
-  // TODO refcount of the strings
   if (st_delete(arg->object_table, &obj, &v)) {
     majo_allocation_info *info = (majo_allocation_info *)v;
     size_t gc_count = rb_gc_count();
-    // Reject it for majo
     if (info->alloc_generation < gc_count-1) {
       info->memsize = rb_obj_memsize_of((VALUE)obj);
       info->free_generation = gc_count;
@@ -110,7 +109,7 @@ start(VALUE self) {
   VALUE stack = rb_ivar_get(rb_mMajo, running_tracer_stack);
   rb_ary_push(stack, res);
 
-  arg->newobj_trace = rb_tracepoint_new(0, RUBY_INTERNAL_EVENT_NEWOBJ, newobj_i, arg);
+  arg->newobj_trace = rb_tracepoint_new(0, RUBY_INTERNAL_EVENT_NEWOBJ, newobj_i, (void *)res);
   arg->freeobj_trace = rb_tracepoint_new(0, RUBY_INTERNAL_EVENT_FREEOBJ, freeobj_i, arg);
 
   rb_tracepoint_enable(arg->newobj_trace);

data/ext/majo/result.c

@@ -7,14 +7,35 @@ static void majo_result_mark(void *ptr)
   rb_gc_mark(arg->freeobj_trace);
 }
 
+static int
+free_st_key(st_data_t key, st_data_t value, st_data_t data)
+{
+    free((void *)key);
+    return ST_CONTINUE;
+}
+
+static int
+free_st_value(st_data_t key, st_data_t value, st_data_t data)
+{
+    free((void *)value);
+    return ST_CONTINUE;
+}
+
 static void majo_result_free(majo_result *arg) {
-  // TODO
-  ruby_xfree(arg);
+  st_foreach(arg->object_table, free_st_value, 0);
+  st_free_table(arg->object_table);
+
+  st_foreach(arg->str_table, free_st_key, 0);
+  st_free_table(arg->str_table);
+
+  rb_darray_free(arg->olds);
+
+  free(arg);
 }
 
 static size_t majo_result_memsize(const void *ptr) {
-  // TODO
-  return sizeof(majo_result);
+  majo_result *res = (majo_result*)ptr;
+  return sizeof(majo_result) + st_memsize(res->object_table) + st_memsize(res->str_table) + rb_darray_capa(res->olds) * sizeof(majo_allocation_info);
 }
 
 static rb_data_type_t result_type = {

data/lib/majo/colorize.rb

@@ -0,0 +1,11 @@
+module Majo
+  module Colorize
+    def blue(str)
+      "\e[34m#{str}\e[0m"
+    end
+
+    def cyan(str)
+      "\e[36m#{str}\e[0m"
+    end
+  end
+end

data/lib/majo/formatter/color.rb

@@ -1,6 +1,8 @@
 module Majo
   module Formatter
     class Color
+      include Colorize
+
       def initialize(result)
         @result = result
       end
@@ -10,21 +12,27 @@ module Majo
           Total #{total_memory} bytes (#{total_objects} objects)
 
           Memory by file
+          #{bar}
           #{format_two_columns(memory_by_file)}
 
           Memory by location
+          #{bar}
           #{format_two_columns(memory_by_location)}
 
           Memory by class
+          #{bar}
           #{format_two_columns(memory_by_class)}
 
           Objects by file
+          #{bar}
           #{format_two_columns(objects_by_file)}
 
           Objects by location
+          #{bar}
           #{format_two_columns(objects_by_location)}
 
           Objects by class
+          #{bar}
           #{format_two_columns(objects_by_class)}
         RESULT
       end
@@ -33,6 +41,10 @@ module Majo
 
       attr_reader :result
 
+      def bar
+        cyan '-----------------------------------'
+      end
+
       def total_objects
         allocs.size
       end
@@ -80,8 +92,8 @@ module Majo
       def format_two_columns(data)
         return "" if data.empty?
 
-        max_length = data.max_by { |row| row[0].to_s.size }[0].size
-        data.map { |row| "#{row[0].to_s.ljust(max_length)} #{row[1]}" }.join("\n")
+        max_length = data.max_by { |row| row[0].to_s.size }[0].to_s.size
+        data.map { |row| "#{blue(row[0].to_s.rjust(max_length))}  #{row[1]}" }.join("\n")
       end
 
       def allocs

data/lib/majo/formatter/monochrome.rb

@@ -0,0 +1,15 @@
+module Majo
+  module Formatter
+    class Monochrome < Color
+      private
+
+      def blue(str)
+        str
+      end
+
+      def cyan(str)
+        str
+      end
+    end
+  end
+end

data/lib/majo/formatter.rb

@@ -1,6 +1,7 @@
 module Majo
   module Formatter
     autoload :Color, 'majo/formatter/color'
+    autoload :Monochrome, 'majo/formatter/monochrome'
     autoload :CSV, 'majo/formatter/csv'
   end
 end

data/lib/majo/result.rb

@@ -3,10 +3,18 @@ module Majo
     def report(out: $stdout, formatter: nil)
       fmt =
         case formatter
-        when nil, :color
+        when :color
           Formatter::Color
         when :csv
           Formatter::CSV
+        when :monochrome
+          Formatter::Monochrome
+        when nil
+          if out.respond_to?(:tty?) && out.tty?
+            Formatter::Color
+          else
+            Formatter::Monochrome
+          end
         else
           raise "unknown formatter: #{formatter.inspect}"
         end

data/lib/majo/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Majo
-  VERSION = "0.0.3"
+  VERSION = "0.1.0"
 end

data/lib/majo.rb

@@ -2,6 +2,7 @@
 
 require_relative 'majo.so'
 require_relative "majo/version"
+require_relative "majo/colorize"
 require_relative 'majo/allocation_info'
 require_relative 'majo/result'
 require_relative 'majo/formatter'

metadata

@@ -1,15 +1,16 @@
 --- !ruby/object:Gem::Specification
 name: majo
 version: !ruby/object:Gem::Version
-  version: 0.0.3
+  version: 0.1.0
 platform: ruby
 authors:
 - Masataka Pocke Kuwabara
+autorequire:
 bindir: exe
 cert_chain: []
-date: 2024-07-22 00:00:00.000000000 Z
+date: 2024-07-25 00:00:00.000000000 Z
 dependencies: []
-description: ''
+description: A memory profiler focusing on long-lived objects.
 email:
 - kuwabara@pocke.me
 executables: []
@@ -19,6 +20,7 @@ extra_rdoc_files: []
 files:
 - ".rspec"
 - ".rubocop.yml"
+- ARCHITECTURE.md
 - LICENSE
 - README.md
 - Rakefile
@@ -34,9 +36,11 @@ files:
 - ext/majo/unique_str.h
 - lib/majo.rb
 - lib/majo/allocation_info.rb
+- lib/majo/colorize.rb
 - lib/majo/formatter.rb
 - lib/majo/formatter/color.rb
 - lib/majo/formatter/csv.rb
+- lib/majo/formatter/monochrome.rb
 - lib/majo/result.rb
 - lib/majo/version.rb
 - sig/majo.rbs
@@ -46,6 +50,7 @@ licenses:
 metadata:
   homepage_uri: https://github.com/pocke/majo
   source_code_uri: https://github.com/pocke/majo
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -60,7 +65,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.6.0.dev
+rubygems_version: 3.5.11
+signing_key:
 specification_version: 4
-summary: ''
+summary: A memory profiler focusing on long-lived objects.
 test_files: []