test-prof-autopilot 0.0.7 → 0.1.0.pre.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: fa21fa398d9ae8f3a79906b3f33d0d75148e9c2ee7ebb5d9900f2302eaefeac1
-  data.tar.gz: cc151f75133d9d71d370c31ba0e18c0466269bfa3241b2adb08005073a981699
+  metadata.gz: 8ee9039d6d83bbf54010e2ebda79747d5ad7511cb7cff65eb1d1064f72561692
+  data.tar.gz: fc3b28fc007dcfefddf66dd4b045befb745db2aa3dc09ca591ae3a75cee45b73
 SHA512:
-  metadata.gz: c4d70b1e42e3aabf5e1c7dc6f9c4403b863e5fc9b95af70cb291d48b1fc4760c91e697b96ff903099a364b3a667e53f151f7934d929e41718a153f88b0a5acc4
-  data.tar.gz: b51e1abe16366626f57c347566d026e504313df6e3b4813b72b95eeaf8ccff9513f7e38729d3ab13fcc1b2afb6756152feb6c5a9b784fbc7433f607b2e7e1585
+  metadata.gz: 9540aa726621514223f0f48a02b27e13c1082b27596093d610b19fd0d885dcb416f6336c03e631fb4d51a10ea57a21ccbb671312b0ca9a02a36a8e73c5225412
+  data.tar.gz: 6d16c822426a62e6ac7ad8d8a0d747d0e61e2850e23252ec9b9753f5ec735cb9b64ff28bb5e071614359a04314ba8eff25d08eb433e4319bfbd53337562808ce

data/LICENSE.txt

@@ -1,6 +1,6 @@
 The MIT License (MIT)
 
-Copyright (c) 2021 Vladimir Dementyev, Ruslan Shakirov
+Copyright (c) 2021-2023 Vladimir Dementyev, Ruslan Shakirov
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

data/README.md

@@ -1,4 +1,7 @@
-# TestProf Autopilot (PoC)
+[![Gem Version](https://badge.fury.io/rb/test-prof-autopilot.svg)](https://rubygems.org/gems/test-prof-autopilot)
+[![Build](https://github.com/test-prof/test-prof-autopilot/workflows/Build/badge.svg)](https://github.com/test-prof/test-prof-autopilot/actions)
+
+# TestProf Autopilot
 
 [TestProf][] has been used by many Ruby/Rails teams to optimize their test suites performance for a while.
 
@@ -6,23 +9,11 @@ Usually, it takes a decent amount of time to profile the test suite initially: w
 
 There are some common patterns in the way we use TestProf, for example: we run StackProf/RubyProf multiple times for different test samples, or we run EventProf for `factory.create` and then use FactoryProf for the slowest tests.
 
-It seems that there is a room for optimization here: we can automate this common tasks, make robots do all the repetition.
-
-This project (codename _TestProf Autopilot_) aims to solve this problem.
-
-## Usage (proposal)
-
-Use a CLI to run a specific tests profiling plan:
-
-```sh
-auto-test-prof -i plan.rb -с "bundle exec rspec"
-```
-
-We specify the base command to run tests via the `-c` option.
+It seems that there is a room for optimization here: we can automate this common tasks, make robots do all the repetition. And here comes **TestProf Autopilot**!
 
-Profiling plan is a Ruby file using a custom DSL to run profilers and access their reports:
+## Usage
 
-Here is an example #2:
+First, write a test profiling plan in a Ruby file. For example, here is how you can perform StackProf profiling multiple times against different random subsets and aggregate the results:
 
 ```ruby
 # This plan runs multiple test samples and collects StackProf data.
@@ -36,41 +27,68 @@ aggregate(3) { run :stackprof, sample: 100 }
 
 # `report` returns the latest generated report (both `run` and `aggregate` set this value automatically)
 # `#methods` returns the list of collected reports sorted by their popularity
-# `info` prints the information (ideally, it should be human-readable)
-info report.methods.take(5)
+puts report.methods.take(5)
 ```
 
-And example #2:
+Now, you can use the `auto-test-prof` command to execute the plan:
 
-```ruby
-# This plan first launch the test suite and collect the information about the time spent in factories.
-# Then it runs FactoryProf for the slowest tests and display the information.
+```sh
+auto-test-prof -i plan.rb -с "bundle exec rspec"
+```
+
+We specify the base command to run tests via the `-c` option. If you omit the command option, Autopilot would fall back to either `bundle exec rspec` or `bundle exec rake test` depending on the presense of the `spec/` and `test/` directories, respectively.
 
-run :event_prof, event: "factory.create"
-run :factory_prof, paths: report.paths
+### Merging results
 
-info report
+Autopilot also allows you to merge reports created with it (using the `#save` method). That's useful when you profile tests on CI and want to see the aggregated results. For example, when using TagProf:
+
+```ruby
+run :tag_prof, events: ["factory.create"]
+
+save report, file_name: "tag_prof_#{ENV["CI_NODE_INDEX"]}"
 ```
 
-### Notes on implementation
+Then, assuming all reports were downloaded:
 
-We use `#run` method to launch tests with profiling. Each profiler has a uniq name (which is the first argument) and some options.
-Some options are common for all profilers (e.g., `sample:` and `paths:`).
+```sh
+$ auto-test-prof --merge tag_prof --reports tag_prof_*.json
 
-## Installation
+Merging tag_prof reports at tag_prof_1.json, tag_prof_2.json, tag_prof_3.json
 
-Adding to a gem:
+[TEST PROF] TagProf report for type
 
-```ruby
-# my-cool-gem.gemspec
-Gem::Specification.new do |spec|
-  # ...
-  spec.add_dependency "test-prof-autopilot"
-  # ...
-end
+       type          time   factory.create    total  %total   %time           avg
+
+      model     28:08.654        19:58.371     1730   56.44   46.23     00:00.976
+    service     20:56.071        16:14.435      808   29.18   28.35     00:01.554
+        api     04:48.179        03:54.178      214    7.32    4.78     00:01.346
+        ...
 ```
 
-Or adding to your project:
+### API
+
+- `run(profiler_name, **options)`: launch the test command with the specified profiler activated; options depend on the profiler, but there are some commont: `sample: <number>` — enables sampling, `paths: <array of paths>` — adds the list of paths to the command.
+
+- `info(report)`: shows the report in the console
+
+- `save(report, path)`: store
+
+- `aggregate(num_calls) { ... }`: aggregates reports obtained by calling the block `num_calls` times.
+
+You can find more examples in the [examples/](examples/) folder.
+
+### Supported profilers
+
+Currently, Autopilot supports the following Test Prof profilers:
+
+- [EventProf](https://test-prof.evilmartians.io/profilers/event_prof) (as `:event_prof`)
+- [TagProf](https://test-prof.evilmartians.io/profilers/tag_prof) (as `:tag_prof`)
+- [StackProf](https://test-prof.evilmartians.io/profilers/stack_prof) (as `:stack_prof`)
+- [FactoryProf](https://test-prof.evilmartians.io/profilers/factory_prof) (as `:factory_prof`)
+
+## Installation
+
+Add the gem to your project:
 
 ```ruby
 # Gemfile
@@ -79,4 +97,8 @@ group :development, :test do
 end
 ```
 
+Make sure `test-prof-autopilot` is required in your test environment.
+
+That's it!
+
 [TestProf]: https://test-prof.evilmartians.io/

data/lib/test-prof-autopilot.rb

@@ -1,3 +1,4 @@
 # frozen_string_literal: true
 
 require "test_prof/autopilot/version"
+require "test_prof/autopilot"

data/lib/test_prof/autopilot/cli.rb

@@ -2,11 +2,14 @@
 
 require "optparse"
 require "test_prof/autopilot/runner"
+require "test_prof/autopilot/logging"
 require "test_prof/autopilot/merger"
 
 module TestProf
   module Autopilot
     class CLI
+      include Logging
+
       attr_reader :command, :plan_path, :mode, :merge_type, :report_paths
 
       def run(args = ARGV)
@@ -15,6 +18,8 @@ module TestProf
         optparser.parse!(args)
 
         if mode == "runner"
+          infer_command! unless command
+
           raise "Test command must be specified. See -h for options" unless command
 
           raise "Plan path must be specified. See -h for options" unless plan_path
@@ -67,6 +72,18 @@ module TestProf
           end
         end
       end
+
+      def infer_command!
+        if Dir.exist?("spec")
+          @command = "bundle exec rspec"
+        elsif Dir.exist?("test")
+          @command = "bundle exec rake test"
+        end
+
+        if @command
+          Logging.log "No command specified, using: #{@command}"
+        end
+      end
     end
   end
 end

data/lib/test_prof/autopilot/command_executor.rb

@@ -7,6 +7,8 @@ module TestProf
     # Module is used for commands execution in child process.
     module CommandExecutor
       def execute(env, command)
+        env.merge!("TEST_PROF_AUTOPILOT_ENABLED" => "true")
+
         Open3.popen2e(env, command) do |_stdin, stdout_and_stderr, _wait_thr|
           while (line = stdout_and_stderr.gets)
             Logging.log line

data/lib/test_prof/autopilot/configuration.rb

@@ -24,8 +24,8 @@ module TestProf
 
       def initialize
         @output = $stdout
-        @tmp_dir = "tmp/test_prof_autopilot"
-        @artifacts_dir = "test_prof_autopilot"
+        @tmp_dir = ENV.fetch("TEST_PROF_AUTOPILOT_TMP_DIR", "tmp/test_prof_autopilot")
+        @artifacts_dir = ENV.fetch("TEST_PROF_AUTOPILOT_DIR", "test_prof_autopilot")
         @merge_format = "info"
       end
     end

data/lib/test_prof/autopilot/event_prof/profiling_executor.rb

@@ -25,6 +25,10 @@ module TestProf
         def build_env
           super.tap do |env|
             env["EVENT_PROF"] = @options[:event]
+            env["EVENT_PROF_TOP"] = @options[:top_count].to_s if @options[:top_count]
+            env["EVENT_PROF_EXAMPLES"] = "1" if @options[:per_example]
+            env["EVENT_PROF_RANK"] = @options[:rank_by].to_s if @options[:rank_by]
+            env["EVENT_PROF_STAMP"] = @options[:stamp] if @options[:stamp]
           end
         end
       end

data/lib/test_prof/autopilot/factory_prof/writer.rb

@@ -1,17 +1,6 @@
 # frozen_string_literal: true
 
 module TestProf
-  using(Module.new do
-    refine FactoryProf::Result do
-      def to_json
-        {
-          stacks: stacks,
-          raw_stats: raw_stats
-        }.to_json
-      end
-    end
-  end)
-
   module Autopilot
     module FactoryProf
       # Class is used for writing :factory_prof report in different formats

data/lib/test_prof/autopilot/merger.rb

@@ -13,10 +13,10 @@ module TestProf
 
       class << self
         def invoke(type, paths)
-          Logging.log "Merging #{type} reports at #{paths.join(", ")}..."
-
           paths = paths.flat_map(&Dir.method(:glob))
 
+          Logging.log "Merging #{type} reports at #{paths.join(", ")}..."
+
           new(type, paths).print_report
         end
       end

data/lib/test_prof/autopilot/stack_prof/report.rb

@@ -24,32 +24,53 @@ module TestProf
         end
 
         def merge(other)
-          f1, f2 = data[:frames], other.data[:frames]
-
-          frames = (f1.keys + f2.keys).uniq.each_with_object({}) do |id, hash|
-            if f1[id].nil?
-              hash[id] = f2[id]
-            elsif f2[id]
-              hash[id] = f1[id]
-              hash[id][:total_samples] += f2[id][:total_samples]
-              hash[id][:samples] += f2[id][:samples]
-              if f2[id][:edges]
-                edges = hash[id][:edges] ||= {}
-                f2[id][:edges].each do |edge, weight|
-                  edges[edge] ||= 0
-                  edges[edge] += weight
-                end
+          ids_mapping = generate_ids_mapping(data[:frames], other.data[:frames])
+
+          frames = data[:frames].dup
+
+          other.data[:frames].each do |id, new_frame|
+            frame =
+              if ids_mapping[id]
+                frames[ids_mapping[id]]
+              else
+                frames[id] = empty_frame_from(new_frame)
               end
-              if f2[id][:lines]
-                lines = hash[id][:lines] ||= {}
-                f2[id][:lines].each do |line, weight|
-                  lines[line] = add_lines(lines[line], weight)
+
+            frame[:total_samples] += new_frame[:total_samples]
+            frame[:samples] += new_frame[:samples]
+
+            if new_frame[:edges]
+              edges = (frame[:edges] ||= {})
+
+              new_frame[:edges].each do |edge, weight|
+                old_edge = ids_mapping[edge]
+
+                if edges[old_edge]
+                  edges[old_edge] += weight
+                else
+                  edges[old_edge] = weight
                 end
               end
-            else
-              hash[id] = f1[id]
             end
-            hash
+
+            if new_frame[:lines]
+              lines = (frame[:lines] ||= {})
+
+              new_frame[:lines].each do |line, weight|
+                old_line = ids_mapping[line]
+
+                lines[old_line] =
+                  if lines[old_line]
+                    add_lines(lines[old_line], weight)
+                  else
+                    weight
+                  end
+              end
+            end
+          end
+
+          converted_raw = other.data[:raw].map do |raw|
+            ids_mapping[raw] || raw
           end
 
           d1, d2 = data, other.data
@@ -62,12 +83,38 @@ module TestProf
             missed_samples: d1[:missed_samples] + d2[:missed_samples],
             metadata: d1[:metadata].merge(d2[:metadata]),
             frames: frames,
-            raw: d1[:raw] + d2[:raw],
+            raw: d1[:raw] + converted_raw,
             raw_timestamp_deltas: d1[:raw_timestamp_deltas] + d2[:raw_timestamp_deltas]
           }
 
           self.class.new(data)
         end
+
+        def generate_ids_mapping(frames, other_frames)
+          old_fingerprints = frames_to_fingerprints(frames)
+          new_fingerprints = frames_to_fingerprints(other_frames)
+
+          new_fingerprints.each_with_object({}) do |(fingerprint, frame), hash|
+            next hash unless old_fingerprints[fingerprint]
+
+            hash[frame[:id]] = old_fingerprints[fingerprint][:id]
+          end
+        end
+
+        def frames_to_fingerprints(frames)
+          frames.each_with_object({}) do |(id, frame), hash|
+            fingerprint = [frame[:name], frame[:file], frame[:line]].compact.map(&:to_s).join("/")
+            hash[fingerprint] = frame.merge(id: id)
+            hash
+          end
+        end
+
+        def empty_frame_from(frame)
+          frame.slice(:name, :file, :line).merge(
+            total_samples: 0,
+            samples: 0
+          )
+        end
       end
     end
   end

data/lib/test_prof/autopilot/version.rb

@@ -2,6 +2,6 @@
 
 module TestProf
   module Autopilot # :nodoc:
-    VERSION = "0.0.7"
+    VERSION = "0.1.0.pre.2"
   end
 end

data/lib/test_prof/autopilot.rb

@@ -2,7 +2,12 @@
 
 require "test-prof"
 require "test_prof/autopilot/configuration"
-require "test_prof/autopilot/patches/event_prof_patch"
-require "test_prof/autopilot/patches/tag_prof_patch"
-require "test_prof/autopilot/patches/factory_prof_patch"
-require "test_prof/autopilot/patches/stack_prof_patch"
+
+# We only load the patches when tests are executed by autopilot
+# TODO: We should move the patches into TestProf itself as `--format=json`.
+if ENV["TEST_PROF_AUTOPILOT_ENABLED"] == "true"
+  require "test_prof/autopilot/patches/event_prof_patch"
+  require "test_prof/autopilot/patches/tag_prof_patch"
+  require "test_prof/autopilot/patches/factory_prof_patch"
+  require "test_prof/autopilot/patches/stack_prof_patch"
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: test-prof-autopilot
 version: !ruby/object:Gem::Version
-  version: 0.0.7
+  version: 0.1.0.pre.2
 platform: ruby
 authors:
 - Ruslan Shakirov
@@ -9,7 +9,7 @@ authors:
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2022-06-14 00:00:00.000000000 Z
+date: 2023-11-15 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: test-prof
@@ -131,7 +131,13 @@ files:
 homepage: http://github.com/test-prof/test-prof-autopilot
 licenses:
 - MIT
-metadata: {}
+metadata:
+  bug_tracker_uri: https://github.com/test-prof/test-prof-autopilot/issues
+  changelog_uri: https://github.com/test-prof/test-prof-autopilot/blob/master/CHANGELOG.md
+  documentation_uri: https://test-prof.evilmartians.io/
+  homepage_uri: https://test-prof.evilmartians.io/
+  source_code_uri: https://github.com/test-prof/test-prof-autopilot
+  funding_uri: https://github.com/sponsors/test-prof
 post_install_message:
 rdoc_options: []
 require_paths:
@@ -143,11 +149,11 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '2.7'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ">="
+  - - ">"
     - !ruby/object:Gem::Version
-      version: '0'
+      version: 1.3.1
 requirements: []
-rubygems_version: 3.3.7
+rubygems_version: 3.4.20
 signing_key:
 specification_version: 4
 summary: Automatic TestProf runner