require-hooks 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 25df07edf75eef2e55e17548e0e02d54ba54e542e4ee3ebe20ec54958d3e2385
-  data.tar.gz: 8337f6fc90e9ee9e917ce30c3f1eb7536309ca590b0882798e70969d519cf626
+  metadata.gz: 0ad50776a8d1a68e226e2813265d503d7e11ecba3a991daf56945a07ca0ae34c
+  data.tar.gz: 906c6874932cfeb3e684b45f572b33f18f25304983a9566c01d540b7570c42ae
 SHA512:
-  metadata.gz: 55979ba3219aacfc8a1626e4c198518f29e527a1bb547e39d86f4fd170d1562c71166df87e50c5d4ad5a7deb58adfa5248dfeac85baa21a41839cecac4441460
-  data.tar.gz: 33c22e744609eccfd80eb46949e7e5740a563cabbbfb9e0a6d68e6fede8a377ea10cfb47998670ae67003e7723e6b2f9c58e3a7cce832488409f254624a4f2f0
+  metadata.gz: 89c6aea0a0fe1c1b1a9dcf4831435d0a6c98d52149ffd8fc079e5fd134a768f8203253b7d5ec9b0148ef5d1f595f19b95274d18f8e889ed4e140a0a1258cc524
+  data.tar.gz: cdc4edf49fc5c7d7f9201e800ee54fda332a1fa94595bc2cf625b0fa032aaf0aa6d72f2653753f399f9bc8ac1d7da9426dab6620197f19f0db8394a390921b9e

data/CHANGELOG.md

@@ -2,6 +2,10 @@
 
 ## master
 
+## 0.2.0 (2023-08-23)
+
+- Add `patterns` and `exclude_patterns` options to hooks. ([@palkan][]])
+
 ## 0.1.0 (2023-07-14)
 
 - Extracted from Ruby Next. ([@palkan][]])

data/README.md

@@ -33,24 +33,29 @@ spec.add_dependency "require-hooks"
 
 ## Usage
 
-To enable hooks, you need to load `require-hooks/setup` as early as possible. For example, in your gem's entrypoint:
+To enable hooks, you need to load `require-hooks/setup` before any code that you want to pre-process via hooks:
 
 ```ruby
 require "require-hooks/setup"
 ```
 
+For example, in an application (e.g., Rails), you may want to only process the source files you own, so you must activate Require Hooks after loading the dependencies (e.g., in the `config/application.rb` file right after `Bundler.require(*)`).
+
+If you want to pre-process all files, you can activate Require Hooks earlier.
+
 Then, you can add hooks:
 
 - **around_load:** a hook that wraps code loading operation. Useful for logging and debugging purposes.
 
 ```ruby
 # Simple logging
-RequireHooks.around_load do |path, &block|
+RequireHooks.around_load(patterns: ["/gem/dir/*.rb"]) do |path, &block|
   puts "Loading #{path}"
   block.call.tap { puts "Loaded #{path}" }
 end
 
-# Error enrichment
+# Error enrichment.
+# No patterns — all files are affected.
 RequireHooks.around_load do |path, &block|
   block.call
 rescue SyntaxError => e
@@ -63,8 +68,7 @@ The return value MUST be a result of calling the passed block.
 - **source_transform:** perform source-to-source transformations.
 
 ```ruby
-RequireHooks.source_transform do |path, source|
-  next unless path =~ /my_project\/.*/
+RequireHooks.source_transform(patterns: ["/my_project/*.rb"], exclude_patterns: ["/my_project/vendor/*"]) do |path, source|
   source ||= File.read(path)
   "# frozen_string_literal: true\n#{source}"
 end
@@ -75,9 +79,8 @@ The return value MUST be either String (new source code) or `nil` (indicating th
 - **hijack_load:** a hook that is used to manually compile byte code for VM to load it.
 
 ```ruby
-RequireHooks.hijack_load do |path, source|
-  next unless path =~ /my_project\/.*/
-
+# Pattern can be a Proc. If it returns `true`, the hijacker is used.
+RequireHooks.hijack_load(patterns: ["/my_project/*.rb"]) do |path, source|
   source ||= File.read(path)
   if defined?(RubyVM::InstructionSequence)
     RubyVM::InstructionSequence.compile(source)
@@ -89,6 +92,8 @@ end
 
 The return value is platform-specific. If there are multiple _hijackers_, the first one that returns a non-`nil` value is used, others are ignored.
 
+**NOTE:** The `patterns` and `exclude_patterns` arguments accept globs as recognized by [File.fnmatch](https://rubyapi.org/3.2/o/file#method-c-fnmatch).
+
 ## Modes
 
 Depending on the runtime conditions, Require Hooks picks an optimal strategy for injecting the code. You can enforce a particular _mode_ by setting the `REQUIRE_HOOKS_MODE` env variable (`patch`, `load_iseq` or `bootsnap`). In practice, only setting to `patch` may makes sense.
@@ -118,6 +123,66 @@ The _around load_ hooks are executed for all files independently of whether they
 
 Thus, if you introduce new source transformers or hijackers, you must invalidate the cache. (We plan to implement automatic invalidation in future versions.)
 
+## Limitations
+
+- `Kernel#load` with a wrap argument (e.g., `load "some_path", true` or `load "some_path", MyModule)`) is not supported (fallbacked to the original implementation). The biggest challenge here is to support constants nesting.
+- Some very edgy symlinking scenarios are not supported (unlikely to affect real-world projects).
+
+## Performance
+
+We conducted a benchmark to measure the performance overhead of Require Hooks using a large Rails project with the following characteristics:
+
+```sh
+$ find config/ lib/ app/ -name "*.rb" | wc -l
+
+2689
+```
+
+```sh
+$ bundle list | wc -l
+
+427
+```
+
+Total number of `#require` calls: **12741**.
+
+We activated Require Hooks in the very start of the program (`config/boot.rb`).
+
+There is a single around load hook to count all the calls:
+
+```ruby
+counter = 0
+RequireHooks.around_load do |_, &block|
+  counter += 1
+  block.call
+end
+
+at_exit { puts "Total hooked calls: #{counter}" }
+```
+
+## Results
+
+All tests made with `eager_load=true`.
+
+Test script: `time bundle exec rails runner 'puts "done"'`.
+
+  |                                   |              |
+|-------------------------------------|--------------|
+| baseline                            |    29s       |
+| baseline w/bootsnap                 |    12s       |
+| rhooks (iseq)                       |    30s       |
+| rhooks (patch)                      |    **8m**    |
+| rhooks (bootsnap)                   |    12s       |
+
+You can see that requiring tons of files with Require Hooks in patch mode is very slow for now. Why? Mostly because we MUST check `$LOADED_FEATURES` for the presence of the file we want to load and currently we do this via `$LOADED_FEATURES.include?(path)` call, which becomes very slow when `$LOADED_FEATURES` is huge. Thus, we recommend activating Require Hooks after loading all the dependencies and limiting the scope of affected files (via the `patterns` option) on non-MRI platforms to avoid this overhead.
+
+**NOTE:** Why Ruby's internal implementations is fast despite from doing the same checks? It uses an internal hash table to keep track of the loaded features (`vm->loaded_features_realpaths`), not an array. Unfortunately, it's not accessible from Ruby.
+
+Here are the numbers for the same project with scoped hooks (only some folders) activated after `Bundler.require(*)`:
+
+- 732 files affected: 2m36s (vs. 30s w/o hooks)
+- 153 files affected: 55s (vs. 30s w/o hooks)
+
 ## Contributing
 
 Bug reports and pull requests are welcome on GitHub at [https://github.com/ruby-next/require-hooks](https://github.com/ruby-next/require-hooks).

data/lib/require-hooks/api.rb

@@ -5,7 +5,61 @@ module RequireHooks
   @@source_transform = []
   @@hijack_load = []
 
+  class Context
+    def initialize(around_load, source_transform, hijack_load)
+      @around_load = around_load
+      @source_transform = source_transform
+      @hijack_load = hijack_load
+    end
+
+    def empty?
+      @around_load.empty? && @source_transform.empty? && @hijack_load.empty?
+    end
+
+    def source_transform?
+      @source_transform.any?
+    end
+
+    def hijack?
+      @hijack_load.any?
+    end
+
+    def run_around_load_callbacks(path)
+      return yield if @around_load.empty?
+
+      chain = @around_load.reverse.inject do |acc_proc, next_proc|
+        proc { |path, &block| acc_proc.call(path) { next_proc.call(path, &block) } }
+      end
+
+      chain.call(path) { yield }
+    end
+
+    def perform_source_transform(path)
+      return unless @source_transform.any?
+
+      source = nil
+
+      @source_transform.each do |transform|
+        source = transform.call(path, source) || source
+      end
+
+      source
+    end
+
+    def try_hijack_load(path, source)
+      return unless @hijack_load.any?
+
+      @hijack_load.each do |hijack|
+        result = hijack.call(path, source)
+        return result if result
+      end
+      nil
+    end
+  end
+
   class << self
+    attr_accessor :print_warnings
+
     # Define a block to wrap the code loading.
     # The return value MUST be a result of calling the passed block.
     # For example, you can use such hooks for instrumentation, debugging purposes.
@@ -14,8 +68,8 @@ module RequireHooks
     #      puts "Loading #{path}"
     #      block.call.tap { puts "Loaded #{path}" }
     #    end
-    def around_load(&block)
-      @@around_load << block
+    def around_load(patterns: nil, exclude_patterns: nil, &block)
+      @@around_load << [patterns, exclude_patterns, block]
     end
 
     # Define hooks to perform source-to-source transformations.
@@ -28,8 +82,8 @@ module RequireHooks
     #    RequireHooks.source_transform do |path, source|
     #      "# frozen_string_literal: true\n#{source}"
     #    end
-    def source_transform(&block)
-      @@source_transform << block
+    def source_transform(patterns: nil, exclude_patterns: nil, &block)
+      @@source_transform << [patterns, exclude_patterns, block]
     end
 
     # This hook should be used to manually compile byte code to be loaded by the VM.
@@ -46,40 +100,33 @@ module RequireHooks
     #       JRuby.compile(source)
     #     end
     #   end
-    def hijack_load(&block)
-      @@hijack_load << block
+    def hijack_load(patterns: nil, exclude_patterns: nil, &block)
+      @@hijack_load << [patterns, exclude_patterns, block]
     end
 
-    def run_around_load_callbacks(path)
-      return yield if @@around_load.empty?
+    def context_for(path)
+      around_load = @@around_load.select do |patterns, exclude_patterns, _block|
+        next unless !patterns || patterns.any? { |pattern| File.fnmatch?(pattern, path) }
+        next if exclude_patterns&.any? { |pattern| File.fnmatch?(pattern, path) }
 
-      chain = @@around_load.reverse.inject do |acc_proc, next_proc|
-        proc { |path, &block| acc_proc.call(path) { next_proc.call(path, &block) } }
-      end
+        true
+      end.map { |_patterns, _exclude_patterns, block| block }
 
-      chain.call(path) { yield }
-    end
+      source_transform = @@source_transform.select do |patterns, exclude_patterns, _block|
+        next unless !patterns || patterns.any? { |pattern| File.fnmatch?(pattern, path) }
+        next if exclude_patterns&.any? { |pattern| File.fnmatch?(pattern, path) }
 
-    def perform_source_transform(path)
-      return unless @@source_transform.any?
+        true
+      end.map { |_patterns, _exclude_patterns, block| block }
 
-      source = nil
+      hijack_load = @@hijack_load.select do |patterns, exclude_patterns, _block|
+        next unless !patterns || patterns.any? { |pattern| File.fnmatch?(pattern, path) }
+        next if exclude_patterns&.any? { |pattern| File.fnmatch?(pattern, path) }
 
-      @@source_transform.each do |transform|
-        source = transform.call(path, source) || source
-      end
-
-      source
-    end
+        true
+      end.map { |_patterns, _exclude_patterns, block| block }
 
-    def try_hijack_load(path, source)
-      return unless @@hijack_load.any?
-
-      @@hijack_load.each do |hijack|
-        result = hijack.call(path, source)
-        return result if result
-      end
-      nil
+      Context.new(around_load, source_transform, hijack_load)
     end
   end
 end

data/lib/require-hooks/mode/bootsnap.rb

@@ -4,8 +4,10 @@ module RequireHooks
   module Bootsnap
     module CompileCacheExt
       def input_to_storage(source, path, *)
-        new_contents = RequireHooks.perform_source_transform(path)
-        hijacked = RequireHooks.try_hijack_load(path, new_contents)
+        ctx = RequireHooks.context_for(path)
+
+        new_contents = ctx.perform_source_transform(path)
+        hijacked = ctx.try_hijack_load(path, new_contents)
 
         if hijacked
           raise TypeError, "Unsupported bytecode format for #{path}: #{hijack.class}" unless hijacked.is_a?(::RubyVM::InstructionSequence)
@@ -24,7 +26,7 @@ module RequireHooks
       # Around hooks must be performed every time we trigger a file load, even if
       # the file is already cached.
       def load_iseq(path)
-        RequireHooks.run_around_load_callbacks(path) { super }
+        RequireHooks.context_for(path).run_around_load_callbacks(path) { super }
       end
     end
   end

data/lib/require-hooks/mode/kernel_patch.rb

@@ -7,9 +7,13 @@ module RequireHooks
   module KernelPatch
     class << self
       def load(path)
-        RequireHooks.run_around_load_callbacks(path) do
-          new_contents = RequireHooks.perform_source_transform(path)
-          hijacked = RequireHooks.try_hijack_load(path, new_contents)
+        ctx = RequireHooks.context_for(path)
+
+        ctx.run_around_load_callbacks(path) do
+          next load_without_require_hooks(path) unless ctx.source_transform? || ctx.hijack?
+
+          new_contents = ctx.perform_source_transform(path)
+          hijacked = ctx.try_hijack_load(path, new_contents)
 
           return try_evaluate(path, hijacked) if hijacked
 
@@ -104,7 +108,7 @@ module RequireHooks
 
           # Recursive require
           if lock.owned? && lock.locked?
-            warn "circular require considered harmful: #{fname}"
+            warn "loading in progress, circular require considered harmful: #{fname}" if RequireHooks.print_warnings
             return yield(true)
           end
 
@@ -227,47 +231,38 @@ module Kernel
     raise TypeError unless path.is_a?(::String)
 
     realpath = nil
+    feature = path
 
     # if extname == ".rb" => lookup feature -> resolve feature -> load
     # if extname != ".rb" => append ".rb" - lookup feature -> resolve feature -> lookup orig (no ext) -> resolve orig (no ext) -> load
     if File.extname(path) != ".rb"
-      return false if RequireHooks::KernelPatch::Features.feature_loaded?(path + ".rb")
-
-      loaded = RequireHooks::KernelPatch::Features::LOCK.lock_feature(path + ".rb") do |loaded|
-        return false if loaded
-
-        realpath = RequireHooks::KernelPatch::Features.feature_path(path + ".rb")
+      realpath = RequireHooks::KernelPatch::Features.feature_path(path + ".rb")
 
-        if realpath
-          $LOADED_FEATURES << realpath
-          RequireHooks::KernelPatch.load(realpath)
-          true
-        end
+      if realpath
+        feature = path + ".rb"
       end
-
-      return true if loaded
     end
 
-    return false if RequireHooks::KernelPatch::Features.feature_loaded?(path)
+    realpath ||= RequireHooks::KernelPatch::Features.feature_path(path)
 
-    loaded = RequireHooks::KernelPatch::Features::LOCK.lock_feature(path) do |loaded|
-      return false if loaded
+    return require_without_require_hooks(path) unless realpath
 
-      realpath = RequireHooks::KernelPatch::Features.feature_path(path)
+    ctx = RequireHooks.context_for(realpath)
 
-      if realpath
-        $LOADED_FEATURES << realpath
-        RequireHooks::KernelPatch.load(realpath)
-        true
-      end
-    end
+    return require_without_require_hooks(path) if ctx.empty?
 
-    return true if loaded
+    return false if RequireHooks::KernelPatch::Features.feature_loaded?(feature)
 
-    require_without_require_hooks(path)
+    RequireHooks::KernelPatch::Features::LOCK.lock_feature(feature) do |loaded|
+      return false if loaded
+
+      $LOADED_FEATURES << realpath
+      RequireHooks::KernelPatch.load(realpath)
+      true
+    end
   rescue LoadError => e
     $LOADED_FEATURES.delete(realpath) if realpath
-    warn "RequireHooks failed to require '#{path}': #{e.message}"
+    warn "RequireHooks failed to require '#{path}': #{e.message}" if RequireHooks.print_warnings
     require_without_require_hooks(path)
   rescue Errno::ENOENT, Errno::EACCES
     raise LoadError, "cannot load such file -- #{path}"
@@ -301,7 +296,7 @@ module Kernel
   alias_method :load_without_require_hooks, :load
   def load(path, wrap = false)
     if wrap
-      warn "RequireHooks does not support `load(smth, wrap: ...)`. Falling back to original `Kernel#load`"
+      warn "RequireHooks does not support `load(smth, wrap: ...)`. Falling back to original `Kernel#load`" if RequireHooks.print_warnings
       return load_without_require_hooks(path, wrap)
     end
 
@@ -325,7 +320,7 @@ module Kernel
   rescue Errno::ENOENT, Errno::EACCES
     raise LoadError, "cannot load such file -- #{path}"
   rescue LoadError => e
-    warn "RuquireHooks failed to load '#{path}': #{e.message}"
+    warn "RuquireHooks failed to load '#{path}': #{e.message}" if RequireHooks.print_warnings
     load_without_require_hooks(path)
   end
 end

data/lib/require-hooks/mode/load_iseq.rb

@@ -3,15 +3,19 @@
 module RequireHooks
   module LoadIseq
     def load_iseq(path)
-      RequireHooks.run_around_load_callbacks(path) do
-        new_contents = RequireHooks.perform_source_transform(path)
-        hijacked = RequireHooks.try_hijack_load(path, new_contents)
+      ctx = RequireHooks.context_for(path)
 
-        if hijacked
-          raise TypeError, "Unsupported bytecode format for #{path}: #{hijack.class}" unless hijacked.is_a?(::RubyVM::InstructionSequence)
-          return hijacked
-        elsif new_contents
-          return RubyVM::InstructionSequence.compile(new_contents, path, path, 1)
+      ctx.run_around_load_callbacks(path) do
+        if ctx.source_transform? || ctx.hijack?
+          new_contents = ctx.perform_source_transform(path)
+          hijacked = ctx.try_hijack_load(path, new_contents)
+
+          if hijacked
+            raise TypeError, "Unsupported bytecode format for #{path}: #{hijack.class}" unless hijacked.is_a?(::RubyVM::InstructionSequence)
+            return hijacked
+          elsif new_contents
+            return RubyVM::InstructionSequence.compile(new_contents, path, path, 1)
+          end
         end
 
         defined?(super) ? super : RubyVM::InstructionSequence.compile_file(path)

data/lib/require-hooks/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module RequireHooks
-  VERSION = "0.1.0"
+  VERSION = "0.2.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: require-hooks
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
 platform: ruby
 authors:
 - Vladimir Dementyev
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2023-07-15 00:00:00.000000000 Z
+date: 2023-08-23 00:00:00.000000000 Z
 dependencies: []
 description: Require Hooks provide infrastructure for intercepting require/load calls
   in Ruby