tapioca 0.5.3 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8201f29f7abb78a5625f206893b18ff2a8f1ac9ffcae2784a90446764680bda3
-  data.tar.gz: 680b85b7a109985b60e4a727fd3295507daec445f26983703c6ba27a2882b24a
+  metadata.gz: a99e7a4475e2991e305c461774e45e128d430632e96581fc541680b61275e4e1
+  data.tar.gz: 8ee7b9b225dd752edf606a98aee1b8490687f351dae642b24060549740d2ab5a
 SHA512:
-  metadata.gz: 0df81ab9e557367adc414569a5bc67e581b2c98a13aa728ba46bcc831dfe490d4c3b81b9fe828c677f9eae74521b94f13c2217014dd900ddbe43144085f1cb83
-  data.tar.gz: 72f0f551108b0d1e2d3b7ce5420ff638fc81bec650a1563b5f0223c04b4646ce625cf5203b047495c873b4cb454b7fd2d910cafffd1afc39b987bac081b7e958
+  metadata.gz: 3ab85b1422fba3aa79de02fad9e164b2013cef3a388ad196e4c11f0c2c6dc142b6a540a430f4a4ace9f3ebe92da2f29fbe733e13ceb481277e7d7e2691db9f0c
+  data.tar.gz: b9f2c2cca77916e08fcbb1ca021818b51bc5edd8d899b2f5e89dae3bfddd38dd8e403b16bacd8c8db37dc630448458d7d6f52e6445cb9ecb96c6bba16c4c20e7

data/README.md

@@ -8,7 +8,7 @@ Tapioca is a library used to generate RBI (Ruby interface) files for use with [S
 
 As yet, no gem exports type information in a consumable format and it would be a huge effort to manually maintain such an interface file for all the gems that your codebase depends on. Thus, there is a need for an automated way to generate the appropriate RBI file for a given gem. The `tapioca` gem, developed at Shopify, is able to do exactly that to almost 99% accuracy. It can generate the definitions for all statically defined types and most of the runtime defined types exported from Ruby gems (non-Ruby gems are not handled yet).
 
-When you run `tapioca sync` in a project, `tapioca` loads all the gems that are in your dependency list from the Gemfile into memory. It then performs runtime introspection on the loaded types to understand their structure and generates an appropriate RBI file for each gem with a versioned filename.
+When you run `tapioca gem` in a project, `tapioca` loads all the gems that are in your dependency list from the Gemfile into memory. It then performs runtime introspection on the loaded types to understand their structure and generates an appropriate RBI file for each gem with a versioned filename.
 
 ## Manual gem requires
 
@@ -33,7 +33,7 @@ from (pry):3:in `__pry__`
 
 In order to make sure that `tapioca` can reflect on that type, we need to add the line `require "better_html/parser"` to the `sorbet/tapioca/require.rb` file. This will make sure `BetterHtml::Parser` is loaded into memory and a type annotation is generated for it in the `better_html.rbi` file. If this extra `require` line is not added to `sorbet/tapioca/require.rb` file, then the definition for that type will be missing from the RBI file.
 
-If you ever run into a case, where you add a gem or update the version of a gem and run `tapioca sync` but don't have some types you expect in the generated gem RBI files, you will need to make sure you have added the necessary requires to the `sorbet/tapioca/require.rb` file.
+If you ever run into a case, where you add a gem or update the version of a gem and run `tapioca gem` but don't have some types you expect in the generated gem RBI files, you will need to make sure you have added the necessary requires to the `sorbet/tapioca/require.rb` file.
 
 You can use the command `tapioca require` to auto-populate the `sorbet/tapioca/require.rb` file with all the requires found
 in your application. Once the file generated, you should review it, remove all unnecessary requires and commit it.
@@ -55,24 +55,21 @@ end
 and do not forget to execute `tapioca` using `bundler`:
 
 ```shell
-$ bundle exec tapioca
+$ bundle exec tapioca help
 Commands:
   tapioca --version, -v      # show version
+  tapioca clean-shims        # clean duplicated definitions in shim RBIs
   tapioca dsl [constant...]  # generate RBIs for dynamic methods
-  tapioca generate [gem...]  # generate RBIs from gems
+  tapioca gem [gem...]       # generate RBIs from gems
   tapioca help [COMMAND]     # Describe available commands or one specific command
   tapioca init               # initializes folder structure
   tapioca require            # generate the list of files to be required by tapioca
-  tapioca sync               # sync RBIs to Gemfile
   tapioca todo               # generate the list of unresolved constants
 
 Options:
-  --pre, -b, [--prerequire=file]                              # A file to be required before Bundler.require is called
-  --post, -a, [--postrequire=file]                            # A file to be required after Bundler.require is called
-  --out, -o, [--outdir=directory]                             # The output directory for generated RBI files
-  --cmd, -c, [--generate-command=command]                     # The command to run to regenerate RBI files
-  -x, [--exclude=gem [gem ...]]                               # Excludes the given gem(s) from RBI generation
-  --typed, -t, [--typed-overrides=gem:level [gem:level ...]]  # Overrides for typed sigils for generated gem RBIs
+  -c, [--config=<config file path>]  # Path to the Tapioca configuration file
+                                     # Default: sorbet/tapioca/config.yml
+  -V, [--verbose], [--no-verbose]    # Verbose output for debugging purposes
 ```
 
 ## Usage
@@ -83,17 +80,54 @@ Command: `tapioca init`
 
 This will create the `sorbet/config` and `sorbet/tapioca/require.rb` files for you, if they don't exist. If any of the files already exist, they will not be changed.
 
-### Generate for gems
+```shell
+$ bundle exec tapioca help init
+Usage:
+  tapioca init
 
-Command: `tapioca generate [gems...]`
+Options:
+  -c, [--config=<config file path>]  # Path to the Tapioca configuration file
+                                     # Default: sorbet/tapioca/config.yml
+  -V, [--verbose], [--no-verbose]    # Verbose output for debugging purposes
 
-This will generate RBIs for the specified gems and place them in the RBI directory.
+initializes folder structure
+```
 
-### Generate for all gems in Gemfile
+### Generate RBI files for gems
 
-Command: `tapioca sync`
+Command: `tapioca gem [gems...]`
 
-This will sync the RBIs with the gems in the Gemfile and will add, update, and remove RBIs as necessary.
+This will generate RBIs for the specified gems and place them in the RBI directory.
+
+```shell
+$ bundle exec tapioca help gem
+Usage:
+  tapioca gem [gem...]
+
+Options:
+  --out, -o, [--outdir=directory]                             # The output directory for generated RBI files
+                                                              # Default: sorbet/rbi/gems
+          [--file-header], [--no-file-header]                 # Add a "This file is generated" header on top of each generated RBI file
+                                                              # Default: true
+          [--all], [--no-all]                                 # Regenerate RBI files for all gems
+  --pre, -b, [--prerequire=file]                              # A file to be required before Bundler.require is called
+  --post, -a, [--postrequire=file]                            # A file to be required after Bundler.require is called
+                                                              # Default: sorbet/tapioca/require.rb
+  -x, [--exclude=gem [gem ...]]                               # Excludes the given gem(s) from RBI generation
+  --typed, -t, [--typed-overrides=gem:level [gem:level ...]]  # Overrides for typed sigils for generated gem RBIs
+                                                              # Default: {"activesupport"=>"false"}
+          [--verify], [--no-verify]                           # Verifies RBIs are up-to-date
+          [--doc], [--no-doc]                                 # Include YARD documentation from sources when generating RBIs. Warning: this might be slow
+          [--exported-gem-rbis], [--no-exported-gem-rbis]     # Include RBIs found in the `rbi/` directory of the gem
+                                                              # Default: true
+  -w, [--workers=N]                                           # EXPERIMENTAL: Number of parallel workers to use when generating RBIs
+                                                              # Default: 1
+  -c, [--config=<config file path>]                           # Path to the Tapioca configuration file
+                                                              # Default: sorbet/tapioca/config.yml
+  -V, [--verbose], [--no-verbose]                             # Verbose output for debugging purposes
+
+generate RBIs from gems
+```
 
 ### Generate the list of all unresolved constants
 
@@ -101,19 +135,76 @@ Command: `tapioca todo`
 
 This will generate the file `sorbet/rbi/todo.rbi` defining all unresolved constants as empty modules.
 
+```shell
+$ bundle exec tapioca help todo
+Usage:
+  tapioca todo
+
+Options:
+      [--todo-file=TODO_FILE]
+                                           # Default: sorbet/rbi/todo.rbi
+      [--file-header], [--no-file-header]  # Add a "This file is generated" header on top of each generated RBI file
+                                           # Default: true
+  -c, [--config=<config file path>]        # Path to the Tapioca configuration file
+                                           # Default: sorbet/tapioca/config.yml
+  -V, [--verbose], [--no-verbose]          # Verbose output for debugging purposes
+
+generate the list of unresolved constants
+```
+
 ### Generate DSL RBI files
 
 Command: `tapioca dsl [constant...]`
 
 This will generate DSL RBIs for specified constants (or for all handled constants, if a constant name is not supplied). You can read about DSL RBI generators supplied by `tapioca` in [the manual](manual/generators.md).
 
-### Flags
+```shell
+$ bundle exec tapioca help dsl
+Usage:
+  tapioca dsl [constant...]
+
+Options:
+  --out, -o, [--outdir=directory]                # The output directory for generated RBI files
+                                                 # Default: sorbet/rbi/dsl
+          [--file-header], [--no-file-header]    # Add a "This file is generated" header on top of each generated RBI file
+                                                 # Default: true
+          [--only=generator [generator ...]]     # Only run supplied DSL generators
+          [--exclude=generator [generator ...]]  # Exclude supplied DSL generators
+          [--verify], [--no-verify]              # Verifies RBIs are up-to-date
+  -q, [--quiet], [--no-quiet]                    # Supresses file creation output
+  -w, [--workers=N]                              # EXPERIMENTAL: Number of parallel workers to use when generating RBIs
+                                                 # Default: 1
+  -c, [--config=<config file path>]              # Path to the Tapioca configuration file
+                                                 # Default: sorbet/tapioca/config.yml
+  -V, [--verbose], [--no-verbose]                # Verbose output for debugging purposes
+
+generate RBIs for dynamic methods
+```
+## Configuration
+
+Tapioca supports loading command defaults from a configuration file. The default configuration
+file location is `sorbet/tapioca/config.yml` but this default can be changed using the `--config` flag
+and supplying an alternative configuration file path.
+
+A configuration file must be a well-formed YAML file with top-level keys for the various Tapioca commands. Keys under each such top-level command should be the underscore version of a long option name for that command and the value for that key should be the value of the option.
+
+For example, if you always want to generate gem RBIs with inline documentation, then you would create the file `sorbet/tapioca/config.yml` as:
 
-- `--prerequire [file]`: A file to be required before `Bundler.require` is called.
-- `--postrequire [file]`: A file to be required after `Bundler.require` is called.
-- `--out [directory]`: The output directory for generated RBI files, default to `sorbet/rbi/gems`.
-- `--generate-command [command]`: **[DEPRECATED]** The command to run to regenerate RBI files (used in header comment of the RBI files), defaults to the current command.
-- `--typed-overrides [gem:level]`: Overrides typed sigils for generated gem RBIs for gem `gem` to level `level` (`level` can be one of `ignore`, `false`, `true`, `strict`, or `strong`, see [the Sorbet docs](https://sorbet.org/docs/static#file-level-granularity-strictness-levels) for more details).
+```yaml
+gem:
+  docs: true
+```
+
+Additionally, if you always want to exclude the `AASM` and `ActiveRecordFixtures` DSL compilers in your DSL RBI generation runs, your config file would then look like this:
+
+```yaml
+gem:
+  docs: true
+dsl:
+  exclude:
+  - UrlHelpers
+  - ActiveRecordFixtures
+```
 
 ## Contributing
 

data/lib/tapioca/cli.rb

@@ -1,46 +1,43 @@
 # typed: true
 # frozen_string_literal: true
 
-require "thor"
-
 module Tapioca
   class Cli < Thor
-    class_option :outdir,
-      aliases: ["--out", "-o"],
-      banner: "directory",
-      desc: "The output directory for generated RBI files"
-    class_option :generate_command,
-      aliases: ["--cmd", "-c"],
-      banner: "command",
-      desc: "The command to run to regenerate RBI files"
-    class_option :file_header,
-      type: :boolean,
-      default: true,
-      desc: "Add a \"This file is generated\" header on top of each generated RBI file"
+    include CliHelper
+    include ConfigHelper
+
+    FILE_HEADER_OPTION_DESC = "Add a \"This file is generated\" header on top of each generated RBI file"
+
+    class_option :config,
+      aliases: ["-c"],
+      banner: "<config file path>",
+      type: :string,
+      desc: "Path to the Tapioca configuration file",
+      default: TAPIOCA_CONFIG_FILE
     class_option :verbose,
       aliases: ["-V"],
       type: :boolean,
-      default: false,
-      desc: "Verbose output for debugging purposes"
-
-    map T.unsafe(["--version", "-v"] => :__print_version)
+      desc: "Verbose output for debugging purposes",
+      default: false
 
     desc "init", "initializes folder structure"
     def init
       generator = Generators::Init.new(
-        sorbet_config: Config::SORBET_CONFIG,
-        default_postrequire: Config::DEFAULT_POSTREQUIRE,
-        default_command: Config::DEFAULT_COMMAND
+        sorbet_config: SORBET_CONFIG_FILE,
+        tapioca_config: TAPIOCA_CONFIG_FILE,
+        default_postrequire: DEFAULT_POSTREQUIRE_FILE,
+        default_command: DEFAULT_COMMAND
       )
       generator.generate
     end
 
     desc "require", "generate the list of files to be required by tapioca"
+    option :postrequire, type: :string, default: DEFAULT_POSTREQUIRE_FILE
     def require
       generator = Generators::Require.new(
-        requires_path: ConfigBuilder.from_options(:require, options).postrequire,
-        sorbet_config_path: Config::SORBET_CONFIG,
-        default_command: Config::DEFAULT_COMMAND
+        requires_path: options[:postrequire],
+        sorbet_config_path: SORBET_CONFIG_FILE,
+        default_command: DEFAULT_COMMAND
       )
       Tapioca.silence_warnings do
         generator.generate
@@ -48,13 +45,18 @@ module Tapioca
     end
 
     desc "todo", "generate the list of unresolved constants"
+    option :todo_file,
+      type: :string,
+      default: DEFAULT_TODO_FILE
+    option :file_header,
+      type: :boolean,
+      desc: FILE_HEADER_OPTION_DESC,
+      default: true
     def todo
-      current_command = T.must(current_command_chain.first)
-      config = ConfigBuilder.from_options(current_command, options)
       generator = Generators::Todo.new(
-        todos_path: config.todos_path,
-        file_header: config.file_header,
-        default_command: Config::DEFAULT_COMMAND
+        todo_file: options[:todo_file],
+        file_header: options[:file_header],
+        default_command: DEFAULT_COMMAND
       )
       Tapioca.silence_warnings do
         generator.generate
@@ -62,15 +64,25 @@ module Tapioca
     end
 
     desc "dsl [constant...]", "generate RBIs for dynamic methods"
-    option :generators,
+    option :outdir,
+      aliases: ["--out", "-o"],
+      banner: "directory",
+      desc: "The output directory for generated DSL RBI files",
+      default: DEFAULT_DSL_DIR
+    option :file_header,
+      type: :boolean,
+      desc: FILE_HEADER_OPTION_DESC,
+      default: true
+    option :only,
       type: :array,
-      aliases: ["--gen", "-g"],
       banner: "generator [generator ...]",
-      desc: "Only run supplied DSL generators"
-    option :exclude_generators,
+      desc: "Only run supplied DSL generator(s)",
+      default: []
+    option :exclude,
       type: :array,
       banner: "generator [generator ...]",
-      desc: "Exclude supplied DSL generators"
+      desc: "Exclude supplied DSL generator(s)",
+      default: []
     option :verify,
       type: :boolean,
       default: false,
@@ -78,75 +90,111 @@ module Tapioca
     option :quiet,
       aliases: ["-q"],
       type: :boolean,
-      desc: "Supresses file creation output"
+      desc: "Supresses file creation output",
+      default: false
+    option :workers,
+      aliases: ["-w"],
+      type: :numeric,
+      desc: "EXPERIMENTAL: Number of parallel workers to use when generating RBIs",
+      default: 1
     def dsl(*constants)
-      current_command = T.must(current_command_chain.first)
-      config = ConfigBuilder.from_options(current_command, options)
       generator = Generators::Dsl.new(
         requested_constants: constants,
-        outpath: config.outpath,
-        generators: config.generators,
-        exclude_generators: config.exclude_generators,
-        file_header: config.file_header,
-        compiler_path: Tapioca::Compilers::Dsl::COMPILERS_PATH,
-        tapioca_path: Config::TAPIOCA_PATH,
-        default_command: Config::DEFAULT_COMMAND,
+        outpath: Pathname.new(options[:outdir]),
+        only: options[:only],
+        exclude: options[:exclude],
+        file_header: options[:file_header],
+        compiler_path: Tapioca::Compilers::Dsl::DSL_COMPILERS_DIR,
+        tapioca_path: TAPIOCA_DIR,
+        default_command: DEFAULT_COMMAND,
         should_verify: options[:verify],
         quiet: options[:quiet],
-        verbose: options[:verbose]
+        verbose: options[:verbose],
+        number_of_workers: options[:workers]
       )
+
+      if options[:workers] != 1
+        say(
+          "Using more than one worker is experimental and might produce results that are not deterministic",
+          :red
+        )
+      end
+
       Tapioca.silence_warnings do
         generator.generate
       end
     end
 
     desc "gem [gem...]", "generate RBIs from gems"
+    option :outdir,
+      aliases: ["--out", "-o"],
+      banner: "directory",
+      desc: "The output directory for generated gem RBI files",
+      default: DEFAULT_GEM_DIR
+    option :file_header,
+      type: :boolean,
+      desc: FILE_HEADER_OPTION_DESC,
+      default: true
     option :all,
       type: :boolean,
-      default: false,
-      desc: "Regenerate RBI files for all gems"
+      desc: "Regenerate RBI files for all gems",
+      default: false
     option :prerequire,
       aliases: ["--pre", "-b"],
       banner: "file",
-      desc: "A file to be required before Bundler.require is called"
+      desc: "A file to be required before Bundler.require is called",
+      default: nil
     option :postrequire,
       aliases: ["--post", "-a"],
       banner: "file",
-      desc: "A file to be required after Bundler.require is called"
+      desc: "A file to be required after Bundler.require is called",
+      default: DEFAULT_POSTREQUIRE_FILE
     option :exclude,
       aliases: ["-x"],
       type: :array,
       banner: "gem [gem ...]",
-      desc: "Excludes the given gem(s) from RBI generation"
+      desc: "Exclude the given gem(s) from RBI generation",
+      default: []
     option :typed_overrides,
       aliases: ["--typed", "-t"],
       type: :hash,
       banner: "gem:level [gem:level ...]",
-      desc: "Overrides for typed sigils for generated gem RBIs"
+      desc: "Override for typed sigils for generated gem RBIs",
+      default: DEFAULT_OVERRIDES
     option :verify,
       type: :boolean,
-      default: false,
-      desc: "Verifies RBIs are up-to-date"
+      desc: "Verify RBIs are up-to-date",
+      default: false
     option :doc,
       type: :boolean,
-      default: false,
-      desc: "Include YARD documentation from sources when generating RBIs. Warning: this might be slow"
+      desc: "Include YARD documentation from sources when generating RBIs. Warning: this might be slow",
+      default: false
+    option :exported_gem_rbis,
+      type: :boolean,
+      desc: "Include RBIs found in the `rbi/` directory of the gem",
+      default: true
+    option :workers,
+      aliases: ["-w"],
+      type: :numeric,
+      desc: "EXPERIMENTAL: Number of parallel workers to use when generating RBIs",
+      default: 1
     def gem(*gems)
       Tapioca.silence_warnings do
         all = options[:all]
         verify = options[:verify]
-        current_command = T.must(current_command_chain.first)
-        config = ConfigBuilder.from_options(current_command, options)
+
         generator = Generators::Gem.new(
           gem_names: all ? [] : gems,
-          gem_excludes: config.exclude,
-          prerequire: config.prerequire,
-          postrequire: config.postrequire,
-          typed_overrides: config.typed_overrides,
-          default_command: Config::DEFAULT_COMMAND,
-          outpath: config.outpath,
-          file_header: config.file_header,
-          doc: config.doc
+          exclude: options[:exclude],
+          prerequire: options[:prerequire],
+          postrequire: options[:postrequire],
+          typed_overrides: options[:typed_overrides],
+          default_command: DEFAULT_COMMAND,
+          outpath: Pathname.new(options[:outdir]),
+          file_header: options[:file_header],
+          doc: options[:doc],
+          include_exported_rbis: options[:exported_gem_rbis],
+          number_of_workers: options[:workers]
         )
 
         raise MalformattedArgumentError, "Options '--all' and '--verify' are mutually exclusive" if all && verify
@@ -156,6 +204,13 @@ module Tapioca
           raise MalformattedArgumentError, "Option '--verify' must be provided without any other arguments" if verify
         end
 
+        if options[:workers] != 1
+          say(
+            "Using more than one worker is experimental and might produce results that are not deterministic",
+            :red
+          )
+        end
+
         if gems.empty? && !all
           generator.sync(should_verify: verify)
         else
@@ -164,6 +219,74 @@ module Tapioca
       end
     end
 
+    desc "clean-shims", "clean duplicated definitions in shim RBIs"
+    option :gem_rbi_dir, type: :string, desc: "Path to gem RBIs", default: DEFAULT_GEM_DIR
+    option :dsl_rbi_dir, type: :string, desc: "Path to DSL RBIs", default: DEFAULT_DSL_DIR
+    option :shim_rbi_dir, type: :string, desc: "Path to shim RBIs", default: DEFAULT_SHIM_DIR
+    def clean_shims(*files_to_clean)
+      index = RBI::Index.new
+
+      # Index gem RBIs
+      gem_rbi_dir = options[:gem_rbi_dir]
+      say("Loading gem RBIs from #{gem_rbi_dir}... ")
+      gem_rbis_files = Dir.glob("#{gem_rbi_dir}/**/*.rbi").sort
+      gem_rbis_trees = RBI::Parser.parse_files(gem_rbis_files)
+      index.visit_all(gem_rbis_trees)
+      say(" Done", :green)
+
+      # Index dsl RBIs
+      dsl_rbi_dir = options[:dsl_rbi_dir]
+      say("Loading dsl RBIs from #{dsl_rbi_dir}... ")
+      dsl_rbis_files = Dir.glob("#{dsl_rbi_dir}/**/*.rbi").sort
+      dsl_rbis_trees = RBI::Parser.parse_files(dsl_rbis_files)
+      index.visit_all(dsl_rbis_trees)
+      say(" Done", :green)
+
+      # Clean shim RBIs
+      if files_to_clean.empty?
+        shim_rbi_dir = options[:shim_rbi_dir]
+        print("Cleaning shim RBIs from #{shim_rbi_dir}...")
+        files_to_clean = Dir.glob("#{shim_rbi_dir}/*.rbi")
+      else
+        print("Cleaning shim RBIs...")
+      end
+
+      done_something = T.let(false, T::Boolean)
+      files_to_clean.sort.each do |path|
+        original = RBI::Parser.parse_file(path)
+        cleaned, operations = RBI::Rewriters::RemoveKnownDefinitions.remove(original, index)
+
+        next if operations.empty?
+        done_something = true
+
+        operations.each do |operation|
+          print("\n  #{operation}")
+        end
+
+        if cleaned.empty?
+          print("\n  Deleted empty file #{path}")
+          FileUtils.rm(path)
+        else
+          File.write(path, cleaned.string)
+        end
+      end
+
+      if done_something
+        say("\nDone", :green)
+      else
+        say(" Done ", :green)
+        say("(nothing to do)", :yellow)
+      end
+    rescue Errno::ENOENT => e
+      say_error("\nCan't read RBI: #{e}")
+      exit(1)
+    rescue RBI::ParseError => e
+      say_error("\nCan't parse RBI: #{e} (#{e.location})")
+      exit(1)
+    end
+
+    map T.unsafe(["--version", "-v"] => :__print_version)
+
     desc "--version, -v", "show version"
     def __print_version
       puts "Tapioca v#{Tapioca::VERSION}"