test-prof-autopilot 0.0.7 → 0.1.0.pre.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: fa21fa398d9ae8f3a79906b3f33d0d75148e9c2ee7ebb5d9900f2302eaefeac1
-  data.tar.gz: cc151f75133d9d71d370c31ba0e18c0466269bfa3241b2adb08005073a981699
+  metadata.gz: 55e6d41b4c30aa3390d97e550a425e92125e6ee74e4ffe6a3e2660be5f5e1795
+  data.tar.gz: 866cc0632a7304a5e675ae78541e77f304de4eeba781f8987a6f8d0563eba3db
 SHA512:
-  metadata.gz: c4d70b1e42e3aabf5e1c7dc6f9c4403b863e5fc9b95af70cb291d48b1fc4760c91e697b96ff903099a364b3a667e53f151f7934d929e41718a153f88b0a5acc4
-  data.tar.gz: b51e1abe16366626f57c347566d026e504313df6e3b4813b72b95eeaf8ccff9513f7e38729d3ab13fcc1b2afb6756152feb6c5a9b784fbc7433f607b2e7e1585
+  metadata.gz: 2ffc441d899c9fa8c2fcaec24930016ee0677705153dd4528aa19e9b48637563a06a0573e001c069d2f04ef5cf4964a49f6380afaefbb263c7b66b0ae8673b19
+  data.tar.gz: 389c5d56dbdc4553d21850f964d5880c0de8a2cde0c9938c65a390de84556b326d3abf8cdc71ad5626461e572e7021697f2272edeb8e34ac1a427f632fafa6f4

data/LICENSE.txt

@@ -1,6 +1,6 @@
 The MIT License (MIT)
 
-Copyright (c) 2021 Vladimir Dementyev, Ruslan Shakirov
+Copyright (c) 2021-2023 Vladimir Dementyev, Ruslan Shakirov
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

data/README.md

@@ -1,4 +1,7 @@
-# TestProf Autopilot (PoC)
+[![Gem Version](https://badge.fury.io/rb/test-prof-autopilot.svg)](https://rubygems.org/gems/test-prof-autopilot)
+[![Build](https://github.com/test-prof/test-prof-autopilot/workflows/Build/badge.svg)](https://github.com/test-prof/test-prof-autopilot/actions)
+
+# TestProf Autopilot
 
 [TestProf][] has been used by many Ruby/Rails teams to optimize their test suites performance for a while.
 
@@ -6,23 +9,11 @@ Usually, it takes a decent amount of time to profile the test suite initially: w
 
 There are some common patterns in the way we use TestProf, for example: we run StackProf/RubyProf multiple times for different test samples, or we run EventProf for `factory.create` and then use FactoryProf for the slowest tests.
 
-It seems that there is a room for optimization here: we can automate this common tasks, make robots do all the repetition.
-
-This project (codename _TestProf Autopilot_) aims to solve this problem.
-
-## Usage (proposal)
-
-Use a CLI to run a specific tests profiling plan:
-
-```sh
-auto-test-prof -i plan.rb -с "bundle exec rspec"
-```
-
-We specify the base command to run tests via the `-c` option.
+It seems that there is a room for optimization here: we can automate this common tasks, make robots do all the repetition. And here comes **TestProf Autopilot**!
 
-Profiling plan is a Ruby file using a custom DSL to run profilers and access their reports:
+## Usage
 
-Here is an example #2:
+First, write a test profiling plan in a Ruby file. For example, here is how you can perform StackProf profiling multiple times against different random subsets and aggregate the results:
 
 ```ruby
 # This plan runs multiple test samples and collects StackProf data.
@@ -36,41 +27,68 @@ aggregate(3) { run :stackprof, sample: 100 }
 
 # `report` returns the latest generated report (both `run` and `aggregate` set this value automatically)
 # `#methods` returns the list of collected reports sorted by their popularity
-# `info` prints the information (ideally, it should be human-readable)
-info report.methods.take(5)
+puts report.methods.take(5)
 ```
 
-And example #2:
+Now, you can use the `auto-test-prof` command to execute the plan:
 
-```ruby
-# This plan first launch the test suite and collect the information about the time spent in factories.
-# Then it runs FactoryProf for the slowest tests and display the information.
+```sh
+auto-test-prof -i plan.rb -с "bundle exec rspec"
+```
+
+We specify the base command to run tests via the `-c` option. If you omit the command option, Autopilot would fall back to either `bundle exec rspec` or `bundle exec rake test` depending on the presense of the `spec/` and `test/` directories, respectively.
 
-run :event_prof, event: "factory.create"
-run :factory_prof, paths: report.paths
+### Merging results
 
-info report
+Autopilot also allows you to merge reports created with it (using the `#save` method). That's useful when you profile tests on CI and want to see the aggregated results. For example, when using TagProf:
+
+```ruby
+run :tag_prof, events: ["factory.create"]
+
+save report, file_name: "tag_prof_#{ENV["CI_NODE_INDEX"]}"
 ```
 
-### Notes on implementation
+Then, assuming all reports were downloaded:
 
-We use `#run` method to launch tests with profiling. Each profiler has a uniq name (which is the first argument) and some options.
-Some options are common for all profilers (e.g., `sample:` and `paths:`).
+```sh
+$ auto-test-prof --merge tag_prof --reports tag_prof_*.json
 
-## Installation
+Merging tag_prof reports at tag_prof_1.json, tag_prof_2.json, tag_prof_3.json
 
-Adding to a gem:
+[TEST PROF] TagProf report for type
 
-```ruby
-# my-cool-gem.gemspec
-Gem::Specification.new do |spec|
-  # ...
-  spec.add_dependency "test-prof-autopilot"
-  # ...
-end
+       type          time   factory.create    total  %total   %time           avg
+
+      model     28:08.654        19:58.371     1730   56.44   46.23     00:00.976
+    service     20:56.071        16:14.435      808   29.18   28.35     00:01.554
+        api     04:48.179        03:54.178      214    7.32    4.78     00:01.346
+        ...
 ```
 
-Or adding to your project:
+### API
+
+- `run(profiler_name, **options)`: launch the test command with the specified profiler activated; options depend on the profiler, but there are some commont: `sample: <number>` — enables sampling, `paths: <array of paths>` — adds the list of paths to the command.
+
+- `info(report)`: shows the report in the console
+
+- `save(report, path)`: store
+
+- `aggregate(num_calls) { ... }`: aggregates reports obtained by calling the block `num_calls` times.
+
+You can find more examples in the [examples/](examples/) folder.
+
+### Supported profilers
+
+Currently, Autopilot supports the following Test Prof profilers:
+
+- [EventProf](https://test-prof.evilmartians.io/profilers/event_prof) (as `:event_prof`)
+- [TagProf](https://test-prof.evilmartians.io/profilers/tag_prof) (as `:tag_prof`)
+- [StackProf](https://test-prof.evilmartians.io/profilers/stack_prof) (as `:stack_prof`)
+- [FactoryProf](https://test-prof.evilmartians.io/profilers/factory_prof) (as `:factory_prof`)
+
+## Installation
+
+Add the gem to your project:
 
 ```ruby
 # Gemfile
@@ -79,4 +97,8 @@ group :development, :test do
 end
 ```
 
+Make sure `test-prof-autopilot` is required in your test environment.
+
+That's it!
+
 [TestProf]: https://test-prof.evilmartians.io/

data/lib/test-prof-autopilot.rb

@@ -1,3 +1,4 @@
 # frozen_string_literal: true
 
 require "test_prof/autopilot/version"
+require "test_prof/autopilot"

data/lib/test_prof/autopilot/cli.rb

@@ -2,11 +2,14 @@
 
 require "optparse"
 require "test_prof/autopilot/runner"
+require "test_prof/autopilot/logging"
 require "test_prof/autopilot/merger"
 
 module TestProf
   module Autopilot
     class CLI
+      include Logging
+
       attr_reader :command, :plan_path, :mode, :merge_type, :report_paths
 
       def run(args = ARGV)
@@ -15,6 +18,8 @@ module TestProf
         optparser.parse!(args)
 
         if mode == "runner"
+          infer_command! unless command
+
           raise "Test command must be specified. See -h for options" unless command
 
           raise "Plan path must be specified. See -h for options" unless plan_path
@@ -67,6 +72,18 @@ module TestProf
           end
         end
       end
+
+      def infer_command!
+        if Dir.exist?("spec")
+          @command = "bundle exec rspec"
+        elsif Dir.exist?("test")
+          @command = "bundle exec rake test"
+        end
+
+        if @command
+          Logging.log "No command specified, using: #{@command}"
+        end
+      end
     end
   end
 end

data/lib/test_prof/autopilot/command_executor.rb

@@ -7,6 +7,8 @@ module TestProf
     # Module is used for commands execution in child process.
     module CommandExecutor
       def execute(env, command)
+        env.merge!("TEST_PROF_AUTOPILOT_ENABLED" => "true")
+
         Open3.popen2e(env, command) do |_stdin, stdout_and_stderr, _wait_thr|
           while (line = stdout_and_stderr.gets)
             Logging.log line

data/lib/test_prof/autopilot/configuration.rb

@@ -24,8 +24,8 @@ module TestProf
 
       def initialize
         @output = $stdout
-        @tmp_dir = "tmp/test_prof_autopilot"
-        @artifacts_dir = "test_prof_autopilot"
+        @tmp_dir = ENV.fetch("TEST_PROF_AUTOPILOT_TMP_DIR", "tmp/test_prof_autopilot")
+        @artifacts_dir = ENV.fetch("TEST_PROF_AUTOPILOT_DIR", "test_prof_autopilot")
         @merge_format = "info"
       end
     end

data/lib/test_prof/autopilot/event_prof/profiling_executor.rb

@@ -25,6 +25,10 @@ module TestProf
         def build_env
           super.tap do |env|
             env["EVENT_PROF"] = @options[:event]
+            env["EVENT_PROF_TOP"] = @options[:top_count].to_s if @options[:top_count]
+            env["EVENT_PROF_EXAMPLES"] = "1" if @options[:per_example]
+            env["EVENT_PROF_RANK"] = @options[:rank_by].to_s if @options[:rank_by]
+            env["EVENT_PROF_STAMP"] = @options[:stamp] if @options[:stamp]
           end
         end
       end

data/lib/test_prof/autopilot/factory_prof/writer.rb

@@ -1,17 +1,6 @@
 # frozen_string_literal: true
 
 module TestProf
-  using(Module.new do
-    refine FactoryProf::Result do
-      def to_json
-        {
-          stacks: stacks,
-          raw_stats: raw_stats
-        }.to_json
-      end
-    end
-  end)
-
   module Autopilot
     module FactoryProf
       # Class is used for writing :factory_prof report in different formats

data/lib/test_prof/autopilot/merger.rb

@@ -13,10 +13,10 @@ module TestProf
 
       class << self
         def invoke(type, paths)
-          Logging.log "Merging #{type} reports at #{paths.join(", ")}..."
-
           paths = paths.flat_map(&Dir.method(:glob))
 
+          Logging.log "Merging #{type} reports at #{paths.join(", ")}..."
+
           new(type, paths).print_report
         end
       end

data/lib/test_prof/autopilot/version.rb

@@ -2,6 +2,6 @@
 
 module TestProf
   module Autopilot # :nodoc:
-    VERSION = "0.0.7"
+    VERSION = "0.1.0.pre.1"
   end
 end

data/lib/test_prof/autopilot.rb

@@ -2,7 +2,12 @@
 
 require "test-prof"
 require "test_prof/autopilot/configuration"
-require "test_prof/autopilot/patches/event_prof_patch"
-require "test_prof/autopilot/patches/tag_prof_patch"
-require "test_prof/autopilot/patches/factory_prof_patch"
-require "test_prof/autopilot/patches/stack_prof_patch"
+
+# We only load the patches when tests are executed by autopilot
+# TODO: We should move the patches into TestProf itself as `--format=json`.
+if ENV["TEST_PROF_AUTOPILOT_ENABLED"] == "true"
+  require "test_prof/autopilot/patches/event_prof_patch"
+  require "test_prof/autopilot/patches/tag_prof_patch"
+  require "test_prof/autopilot/patches/factory_prof_patch"
+  require "test_prof/autopilot/patches/stack_prof_patch"
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: test-prof-autopilot
 version: !ruby/object:Gem::Version
-  version: 0.0.7
+  version: 0.1.0.pre.1
 platform: ruby
 authors:
 - Ruslan Shakirov
@@ -9,7 +9,7 @@ authors:
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2022-06-14 00:00:00.000000000 Z
+date: 2023-11-15 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: test-prof
@@ -131,7 +131,13 @@ files:
 homepage: http://github.com/test-prof/test-prof-autopilot
 licenses:
 - MIT
-metadata: {}
+metadata:
+  bug_tracker_uri: https://github.com/test-prof/test-prof-autopilot/issues
+  changelog_uri: https://github.com/test-prof/test-prof-autopilot/blob/master/CHANGELOG.md
+  documentation_uri: https://test-prof.evilmartians.io/
+  homepage_uri: https://test-prof.evilmartians.io/
+  source_code_uri: https://github.com/test-prof/test-prof-autopilot
+  funding_uri: https://github.com/sponsors/test-prof
 post_install_message:
 rdoc_options: []
 require_paths:
@@ -143,11 +149,11 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '2.7'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ">="
+  - - ">"
     - !ruby/object:Gem::Version
-      version: '0'
+      version: 1.3.1
 requirements: []
-rubygems_version: 3.3.7
+rubygems_version: 3.4.20
 signing_key:
 specification_version: 4
 summary: Automatic TestProf runner