bootsnap 1.18.6 → 1.20.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 962bb65ac06446a7f6a1bace4373f552c46f198cedbf32fa1584a531c7f96f48
-  data.tar.gz: fc84432aa3a728562b7a3d0b29d2edf74a2ee39a888129e56882e8c061571a2d
+  metadata.gz: 8e0a92c2964b6f614db5d8bf1636026c49796a74ae82b1cf78d2e3ec39337fa2
+  data.tar.gz: 8f2e985c4af37b7056b0ddbee9fdd98b31839eb50ce2bf028d3089d92c6a4a88
 SHA512:
-  metadata.gz: 18daeec63113ff6eefc97058720d2e9d1703da5e2cc5d1a7e69db3d3d927a89ac156164138c113bcd44dd1fe4f49e7177b4f552c4839872fa11a6385e7413655
-  data.tar.gz: a8b29f7ea54243953887da78816599bec864240650072e53496446146190c9be8f54a2539cd29f4aebcccb0cbcacdad2dfd2ae187e61b6dcad0ea91fe6aeddcc
+  metadata.gz: e539a95b7f098a337b97a91cd9c29849b2734fac66716cb46b29bcb8e67c7fddd6b0ca07f3a0a6d9ecf6dca2d28b99f0ee5ae9743193f1b7f3ef0994117391d8
+  data.tar.gz: 56b9d42182d85530439defae95000e4aceec364e6ba4a1e69815127a3a5c782b330cef032c5d6f0ceb9b19ca2da30c6bc192eac33c878664e878772134137545

data/CHANGELOG.md

@@ -1,5 +1,15 @@
 # Unreleased
 
+# 1.20.0
+
+* Optimized load path scanning with a C extension. Should be about 2x faster on supported platforms.
+
+# 1.19.0
+
+* Remove JSON parsing cache. Recent versions of the `json` gem are as fast as `msgpack` if not faster.
+
+# 1.18.6
+
 * Fix cgroup CPU limits detection in CLI.
 
 # 1.18.5
@@ -103,7 +113,7 @@
 
 * Get rid of the `Kernel.require_relative` decorator by resolving `$LOAD_PATH` members to their real path.
   This way we handle symlinks in `$LOAD_PATH` much more efficiently. See #402 for the detailed explanation.
-  
+
 * Drop support for Ruby 2.3 (to allow getting rid of the `Kernel.require_relative` decorator).
 
 # 1.10.3
@@ -229,7 +239,7 @@
 * Adds an instrumentation API to monitor cache misses.
 * Allow to control the behavior of `require 'bootsnap/setup'` using environment variables.
 * Deprecate the `disable_trace` option.
-* Deprecate the `ActiveSupport::Dependencies` (AKA Classic autoloader) integration. (#344) 
+* Deprecate the `ActiveSupport::Dependencies` (AKA Classic autoloader) integration. (#344)
 
 # 1.6.0
 
@@ -249,12 +259,12 @@
 
 # 1.4.9
 
-* [Windows support](https://github.com/Shopify/bootsnap/pull/319)
-* [Fix potential crash](https://github.com/Shopify/bootsnap/pull/322)
+* [Windows support](https://github.com/rails/bootsnap/pull/319)
+* [Fix potential crash](https://github.com/rails/bootsnap/pull/322)
 
 # 1.4.8
 
-* [Prevent FallbackScan from polluting exception cause](https://github.com/Shopify/bootsnap/pull/314)
+* [Prevent FallbackScan from polluting exception cause](https://github.com/rails/bootsnap/pull/314)
 
 # 1.4.7
 
@@ -267,7 +277,7 @@
   required if a different file with the same name was already being required
 
   Example:
-  
+
       require 'foo'
       require 'foo.en'
 
@@ -321,7 +331,7 @@
 
 # 1.3.0
 
-* Handle cases where load path entries are symlinked (https://github.com/Shopify/bootsnap/pull/136)
+* Handle cases where load path entries are symlinked (https://github.com/rails/bootsnap/pull/136)
 
 # 1.2.1
 

data/LICENSE.txt

@@ -1,6 +1,7 @@
 The MIT License (MIT)
 
-Copyright (c) 2017-present Shopify, Inc.
+Copyright (c) 2017-2025 Shopify, Inc.
+Copyright (c) 2025-present Rails Foundation
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

data/README.md

@@ -1,4 +1,4 @@
-# Bootsnap [![Actions Status](https://github.com/Shopify/bootsnap/workflows/ci/badge.svg)](https://github.com/Shopify/bootsnap/actions)
+# Bootsnap [![Actions Status](https://github.com/rails/bootsnap/workflows/ci/badge.svg)](https://github.com/rails/bootsnap/actions)
 
 Bootsnap is a library that plugs into Ruby, with optional support for `YAML` and `JSON`,
 to optimize and cache expensive computations. See [How Does This Work](#how-does-this-work).
@@ -41,7 +41,7 @@ getting progressively slower, this is almost certainly the cause.**
 It's technically possible to simply specify `gem 'bootsnap', require: 'bootsnap/setup'`, but it's
 important to load Bootsnap as early as possible to get maximum performance improvement.
 
-You can see how this require works [here](https://github.com/Shopify/bootsnap/blob/main/lib/bootsnap/setup.rb).
+You can see how this require works [here](https://github.com/rails/bootsnap/blob/main/lib/bootsnap/setup.rb).
 
 If you are not using Rails, or if you are but want more control over things, add this to your
 application setup immediately after `require 'bundler/setup'` (i.e. as early as possible: the sooner
@@ -57,13 +57,12 @@ Bootsnap.setup(
   load_path_cache:      true,                 # Optimize the LOAD_PATH with a cache
   compile_cache_iseq:   true,                 # Compile Ruby code into ISeq cache, breaks coverage reporting.
   compile_cache_yaml:   true,                 # Compile YAML into a cache
-  compile_cache_json:   true,                 # Compile JSON into a cache
   readonly:             true,                 # Use the caches but don't update them on miss or stale entries.
 )
 ```
 
 **Protip:** You can replace `require 'bootsnap'` with `BootLib::Require.from_gem('bootsnap',
-'bootsnap')` using [this trick](https://github.com/Shopify/bootsnap/wiki/Bootlib::Require). This
+'bootsnap')` using [this trick](https://github.com/rails/bootsnap/wiki/Bootlib::Require). This
 will help optimize boot time further if you have an extremely large `$LOAD_PATH`.
 
 Note: Bootsnap and [Spring](https://github.com/rails/spring) are orthogonal tools. While Bootsnap
@@ -170,7 +169,7 @@ The only directories considered "stable" are things under the Ruby install prefi
 "volatile".
 
 In addition to the [`Bootsnap::LoadPathCache::Cache`
-source](https://github.com/Shopify/bootsnap/blob/main/lib/bootsnap/load_path_cache/cache.rb),
+source](https://github.com/rails/bootsnap/blob/main/lib/bootsnap/load_path_cache/cache.rb),
 this diagram may help clarify how entry resolution works:
 
 ![How path searching works](https://cloud.githubusercontent.com/assets/3074765/25388270/670b5652-299b-11e7-87fb-975647f68981.png)
@@ -334,6 +333,20 @@ Example:
 $ bundle exec bootsnap precompile --gemfile app/ lib/ config/
 ```
 
+## Known issues
+
+### QEMU environments
+
+When building cross-platform Docker images, QEMU is often used for emulation and can be the source of a limitation that causes forked processes to hang. While Bootsnap includes automatic detection for this issue (as of [PR #501](https://github.com/rails/bootsnap/pull/501)), the detection may not always be sufficient.
+
+If you encounter hangs during precompilation in QEMU-based environments (such as when using Docker buildx for cross-platform builds), you can work around this by disabling parallelization with the `-j 0` option:
+
+```bash
+$ bundle exec bootsnap precompile -j 0 --gemfile app/ lib/ config/
+```
+
+See [Issue #495](https://github.com/rails/bootsnap/issues/495) for more details about this QEMU-related issue.
+
 ## When not to use Bootsnap
 
 *Alternative engines*: Bootsnap is pretty reliant on MRI features, and parts are disabled entirely on alternative ruby

data/ext/bootsnap/bootsnap.c

@@ -11,7 +11,6 @@
  * here.
  */
 
-#include "bootsnap.h"
 #include "ruby.h"
 #include <stdint.h>
 #include <stdbool.h>
@@ -20,10 +19,11 @@
 #include <fcntl.h>
 #include <unistd.h>
 #include <sys/stat.h>
+#include <dirent.h>
 
 #ifdef __APPLE__
   // The symbol is present, however not in the headers
-  // See: https://github.com/Shopify/bootsnap/issues/470
+  // See: https://github.com/rails/bootsnap/issues/470
   extern int fdatasync(int);
 #endif
 
@@ -96,7 +96,6 @@ static mode_t current_umask;
 
 /* Bootsnap::CompileCache::{Native, Uncompilable} */
 static VALUE rb_mBootsnap;
-static VALUE rb_mBootsnap_CompileCache;
 static VALUE rb_mBootsnap_CompileCache_Native;
 static VALUE rb_cBootsnap_CompileCache_UNCOMPILABLE;
 static ID instrumentation_method;
@@ -152,6 +151,84 @@ bs_rb_get_path(VALUE self, VALUE fname)
     return rb_get_path(fname);
 }
 
+#ifdef HAVE_FSTATAT
+static VALUE
+bs_rb_scan_dir(VALUE self, VALUE abspath)
+{
+    Check_Type(abspath, T_STRING);
+
+    DIR *dirp = opendir(RSTRING_PTR(abspath));
+
+    VALUE dirs = rb_ary_new();
+    VALUE requirables = rb_ary_new();
+    VALUE result = rb_ary_new_from_args(2, requirables, dirs);
+
+    if (dirp == NULL) {
+        if (errno == ENOTDIR || errno == ENOENT) {
+            return result;
+        }
+        rb_sys_fail("opendir");
+        return Qundef;
+    }
+
+    struct dirent *entry;
+    struct stat st;
+    int dfd = -1;
+
+    errno = 0;
+    while ((entry = readdir(dirp))) {
+        if (entry->d_name[0] == '.') continue;
+
+        if (RB_UNLIKELY(entry->d_type == DT_UNKNOWN || entry->d_type == DT_LNK)) {
+            // Note: the original implementation of LoadPathCache did follow symlink.
+            // So this is replicated here, but I'm not sure it's a good idea.
+            if (dfd < 0) {
+                dfd = dirfd(dirp);
+                if (dfd < 0) {
+                    rb_sys_fail("dirfd");
+                    return Qundef;
+                }
+            }
+
+            if (fstatat(dfd, entry->d_name, &st, 0)) {
+                rb_sys_fail("fstatat");
+                return Qundef;
+            }
+
+            if (S_ISREG(st.st_mode)) {
+                entry->d_type = DT_REG;
+            } else if (S_ISDIR(st.st_mode)) {
+                entry->d_type = DT_DIR;
+            }
+        }
+
+        if (entry->d_type == DT_DIR) {
+            rb_ary_push(dirs, rb_utf8_str_new_cstr(entry->d_name));
+            continue;
+        } else if (entry->d_type == DT_REG) {
+            size_t len = strlen(entry->d_name);
+            bool is_requirable = (
+                // Comparing 4B allows compiler to optimize this into a single 32b integer comparison.
+                (len > 3 && memcmp(entry->d_name + (len - 3), ".rb", 4) == 0)
+                || (len > DLEXT_MAXLEN && memcmp(entry->d_name + (len - DLEXT_MAXLEN), DLEXT, DLEXT_MAXLEN + 1) == 0)
+#ifdef DLEXT2
+                || (len > DLEXT2_MAXLEN && memcmp(entry->d_name + (len - DLEXT2_MAXLEN), DLEXT2, DLEXT2_MAXLEN + 1) == 0)
+#endif
+            );
+            if (is_requirable) {
+                rb_ary_push(requirables, rb_utf8_str_new(entry->d_name, len));
+            }
+        }
+    }
+
+    if (closedir(dirp)) {
+        rb_sys_fail("closedir");
+        return Qundef;
+    }
+    return result;
+}
+#endif
+
 /*
  * Ruby C extensions are initialized by calling Init_<extname>.
  *
@@ -166,7 +243,14 @@ Init_bootsnap(void)
 
   rb_define_singleton_method(rb_mBootsnap, "rb_get_path", bs_rb_get_path, 1);
 
-  rb_mBootsnap_CompileCache = rb_define_module_under(rb_mBootsnap, "CompileCache");
+#ifdef HAVE_FSTATAT
+  VALUE rb_mBootsnap_LoadPathCache = rb_define_module_under(rb_mBootsnap, "LoadPathCache");
+  VALUE rb_mBootsnap_LoadPathCache_Native = rb_define_module_under(rb_mBootsnap_LoadPathCache, "Native");
+
+  rb_define_singleton_method(rb_mBootsnap_LoadPathCache_Native, "scan_dir", bs_rb_scan_dir, 1);
+#endif
+
+  VALUE rb_mBootsnap_CompileCache = rb_define_module_under(rb_mBootsnap, "CompileCache");
   rb_mBootsnap_CompileCache_Native = rb_define_module_under(rb_mBootsnap_CompileCache, "Native");
   rb_cBootsnap_CompileCache_UNCOMPILABLE = rb_const_get(rb_mBootsnap_CompileCache, rb_intern("UNCOMPILABLE"));
   rb_global_variable(&rb_cBootsnap_CompileCache_UNCOMPILABLE);

data/ext/bootsnap/extconf.rb

@@ -4,6 +4,7 @@ require "mkmf"
 
 if %w[ruby truffleruby].include?(RUBY_ENGINE)
   have_func "fdatasync", "unistd.h"
+  have_func "fstatat", "sys/stat.h"
 
   unless RUBY_PLATFORM.match?(/mswin|mingw|cygwin/)
     append_cppflags ["-D_GNU_SOURCE"] # Needed of O_NOATIME
@@ -12,7 +13,7 @@ if %w[ruby truffleruby].include?(RUBY_ENGINE)
   append_cflags ["-O3", "-std=c99"]
 
   # ruby.h has some -Wpedantic fails in some cases
-  # (e.g. https://github.com/Shopify/bootsnap/issues/15)
+  # (e.g. https://github.com/rails/bootsnap/issues/15)
   unless ["0", "", nil].include?(ENV["BOOTSNAP_PEDANTIC"])
     append_cflags([
       "-Wall",

data/lib/bootsnap/cli/worker_pool.rb

@@ -59,7 +59,7 @@ module Bootsnap
         def fork_defunct?
           return true unless ::Process.respond_to?(:fork)
 
-          # Ref: https://github.com/Shopify/bootsnap/issues/495
+          # Ref: https://github.com/rails/bootsnap/issues/495
           # The second forked process will hang on some QEMU environments
           r, w = IO.pipe
           pids = 2.times.map do

data/lib/bootsnap/cli.rb

@@ -20,7 +20,7 @@ module Bootsnap
 
     attr_reader :cache_dir, :argv
 
-    attr_accessor :compile_gemfile, :exclude, :verbose, :iseq, :yaml, :json, :jobs
+    attr_accessor :compile_gemfile, :exclude, :verbose, :iseq, :yaml, :jobs
 
     def initialize(argv)
       @argv = argv
@@ -31,7 +31,6 @@ module Bootsnap
       self.jobs = nil
       self.iseq = true
       self.yaml = true
-      self.json = true
     end
 
     def precompile_command(*sources)
@@ -42,21 +41,18 @@ module Bootsnap
           cache_dir: cache_dir,
           iseq: iseq,
           yaml: yaml,
-          json: json,
           revalidation: true,
         )
 
         @work_pool = WorkerPool.create(size: jobs, jobs: {
           ruby: method(:precompile_ruby),
           yaml: method(:precompile_yaml),
-          json: method(:precompile_json),
         })
         @work_pool.spawn
 
         main_sources = sources.map { |d| File.expand_path(d) }
         precompile_ruby_files(main_sources)
         precompile_yaml_files(main_sources)
-        precompile_json_files(main_sources)
 
         if compile_gemfile
           # Gems that include JSON or YAML files usually don't put them in `lib/`.
@@ -70,7 +66,6 @@ module Bootsnap
 
           precompile_ruby_files(gem_paths, exclude: gem_exclude)
           precompile_yaml_files(gem_paths, exclude: gem_exclude)
-          precompile_json_files(gem_paths, exclude: gem_exclude)
         end
 
         if (exitstatus = @work_pool.shutdown)
@@ -145,29 +140,6 @@ module Bootsnap
       end
     end
 
-    def precompile_json_files(load_paths, exclude: self.exclude)
-      return unless json
-
-      load_paths.each do |path|
-        if !exclude || !exclude.match?(path)
-          list_files(path, "**/*.json").each do |json_file|
-            # We ignore hidden files to not match the various .config.json files
-            if !File.basename(json_file).start_with?(".") && (!exclude || !exclude.match?(json_file))
-              @work_pool.push(:json, json_file)
-            end
-          end
-        end
-      end
-    end
-
-    def precompile_json(*json_files)
-      Array(json_files).each do |json_file|
-        if CompileCache::JSON.precompile(json_file) && verbose
-          $stderr.puts(json_file)
-        end
-      end
-    end
-
     def precompile_ruby_files(load_paths, exclude: self.exclude)
       return unless iseq
 
@@ -276,9 +248,9 @@ module Bootsnap
         opts.on("--no-yaml", help) { self.yaml = false }
 
         help = <<~HELP
-          Disable JSON precompilation.
+          Disable JSON precompilation. Deprecated.
         HELP
-        opts.on("--no-json", help) { self.json = false }
+        opts.on("--no-json", help) { $stderr.puts("The --no-json option is deprecated and now a noop.") }
       end
     end
   end

data/lib/bootsnap/compile_cache.rb

@@ -9,7 +9,11 @@ module Bootsnap
 
     Error = Class.new(StandardError)
 
-    def self.setup(cache_dir:, iseq:, yaml:, json:, readonly: false, revalidation: false)
+    def self.setup(cache_dir:, iseq:, yaml:, json: (json_unset = true), readonly: false, revalidation: false)
+      unless json_unset
+        warn("Bootsnap::CompileCache.setup `json` argument is deprecated and has no effect")
+      end
+
       if iseq
         if supported?
           require_relative "compile_cache/iseq"
@@ -28,15 +32,6 @@ module Bootsnap
         end
       end
 
-      if json
-        if supported?
-          require_relative "compile_cache/json"
-          Bootsnap::CompileCache::JSON.install!(cache_dir)
-        elsif $VERBOSE
-          warn("[bootsnap/setup] JSON parsing caching is not supported on this implementation of Ruby")
-        end
-      end
-
       if supported? && defined?(Bootsnap::CompileCache::Native)
         Bootsnap::CompileCache::Native.readonly = readonly
         Bootsnap::CompileCache::Native.revalidation = revalidation

data/lib/bootsnap/load_path_cache/cache.rb

@@ -11,19 +11,19 @@ module Bootsnap
         @development_mode = development_mode
         @store = store
         @mutex = Mutex.new
-        @path_obj = path_obj.map! { |f| PathScanner.os_path(File.exist?(f) ? File.realpath(f) : f.dup) }
+        @path_obj = path_obj.map! do |f|
+          if File.exist?(f)
+            File.realpath(f).freeze
+          elsif f.frozen?
+            f
+          else
+            f.dup.freeze
+          end
+        end
         @has_relative_paths = nil
         reinitialize
       end
 
-      # What is the path item that contains the dir as child?
-      # e.g. given "/a/b/c/d" exists, and the path is ["/a/b"], load_dir("c/d")
-      # is "/a/b".
-      def load_dir(dir)
-        reinitialize if stale?
-        @mutex.synchronize { @dirs[dir] }
-      end
-
       TRUFFLERUBY_LIB_DIR_PREFIX = if RUBY_ENGINE == "truffleruby"
         "#{File.join(RbConfig::CONFIG['libdir'], 'truffle')}#{File::SEPARATOR}"
       end
@@ -124,7 +124,6 @@ module Bootsnap
           @path_obj = path_obj
           ChangeObserver.register(@path_obj, self)
           @index = {}
-          @dirs = {}
           @generated_at = now
           push_paths_locked(*@path_obj)
         end
@@ -152,9 +151,8 @@ module Bootsnap
             p = p.to_realpath
 
             expanded_path = p.expanded_path
-            entries, dirs = p.entries_and_dirs(@store)
+            entries = p.entries(@store)
             # push -> low precedence -> set only if unset
-            dirs.each    { |dir| @dirs[dir] ||= path }
             entries.each { |rel| @index[rel] ||= expanded_path }
           end
         end
@@ -169,9 +167,8 @@ module Bootsnap
             p = p.to_realpath
 
             expanded_path = p.expanded_path
-            entries, dirs = p.entries_and_dirs(@store)
+            entries = p.entries(@store)
             # unshift -> high precedence -> unconditional set
-            dirs.each    { |dir| @dirs[dir]  = path }
             entries.each { |rel| @index[rel] = expanded_path }
           end
         end

data/lib/bootsnap/load_path_cache/path.rb

@@ -54,29 +54,28 @@ module Bootsnap
         !path.start_with?(SLASH)
       end
 
-      # Return a list of all the requirable files and all of the subdirectories
-      # of this +Path+.
-      def entries_and_dirs(store)
+      # Return a list of all the requirable files of this +Path+.
+      def entries(store)
         if stable?
           # the cached_mtime field is unused for 'stable' paths, but is
           # set to zero anyway, just in case we change the stability heuristics.
-          _, entries, dirs = store.get(expanded_path)
-          return [entries, dirs] if entries # cache hit
+          _, entries, = store.get(expanded_path)
+          return entries if entries # cache hit
 
-          entries, dirs = scan!
-          store.set(expanded_path, [0, entries, dirs])
-          return [entries, dirs]
+          entries = PathScanner.call(expanded_path)
+          store.set(expanded_path, [0, entries])
+          return entries
         end
 
-        cached_mtime, entries, dirs = store.get(expanded_path)
+        cached_mtime, entries = store.get(expanded_path)
 
-        current_mtime = latest_mtime(expanded_path, dirs || [])
-        return [[], []]        if current_mtime == -1 # path does not exist
-        return [entries, dirs] if cached_mtime == current_mtime
+        current_mtime = latest_mtime(expanded_path, entries || [])
+        return [] if current_mtime == -1 # path does not exist
+        return entries if cached_mtime == current_mtime
 
-        entries, dirs = scan!
-        store.set(expanded_path, [current_mtime, entries, dirs])
-        [entries, dirs]
+        entries = PathScanner.call(expanded_path)
+        store.set(expanded_path, [current_mtime, entries])
+        entries
       end
 
       def expanded_path
@@ -89,23 +88,31 @@ module Bootsnap
 
       private
 
-      def scan! # (expensive) returns [entries, dirs]
-        PathScanner.call(expanded_path)
-      end
-
       # last time a directory was modified in this subtree. +dirs+ should be a
       # list of relative paths to directories under +path+. e.g. for /a/b and
       # /a/b/c, pass ('/a/b', ['c'])
-      def latest_mtime(path, dirs)
-        max = -1
-        ["", *dirs].each do |dir|
-          curr = begin
-            File.mtime("#{path}/#{dir}").to_i
-                 rescue Errno::ENOENT, Errno::ENOTDIR, Errno::EINVAL
-                   -1
+      def latest_mtime(root_path, entries)
+        max = begin
+          File.mtime(root_path).to_i
+        rescue Errno::ENOENT, Errno::ENOTDIR, Errno::EINVAL
+          -1
+        end
+
+        visited = {"." => true}
+
+        entries.each do |relpath|
+          dirname = File.dirname(relpath)
+          visited[dirname] ||= begin
+            begin
+              current = File.mtime(File.join(root_path, dirname)).to_i
+              max = current if current > max
+            rescue Errno::ENOENT, Errno::ENOTDIR, Errno::EINVAL
+              # ignore
+            end
+            true
           end
-          max = curr if curr > max
         end
+
         max
       end
 

data/lib/bootsnap/load_path_cache/path_scanner.rb

@@ -6,8 +6,6 @@ module Bootsnap
   module LoadPathCache
     module PathScanner
       REQUIRABLE_EXTENSIONS = [DOT_RB] + DL_EXTENSIONS
-      NORMALIZE_NATIVE_EXTENSIONS = !DL_EXTENSIONS.include?(LoadPathCache::DOT_SO)
-      ALTERNATIVE_NATIVE_EXTENSIONS_PATTERN = /\.(o|bundle|dylib)\z/.freeze
 
       BUNDLE_PATH = if Bootsnap.bundler?
         (Bundler.bundle_path.cleanpath.to_s << LoadPathCache::SLASH).freeze
@@ -20,9 +18,9 @@ module Bootsnap
       class << self
         attr_accessor :ignored_directories
 
-        def call(path)
+        def ruby_call(path)
           path = File.expand_path(path.to_s).freeze
-          return [[], []] unless File.directory?(path)
+          return [] unless File.directory?(path)
 
           # If the bundle path is a descendent of this path, we do additional
           # checks to prevent recursing into the bundle path as we recurse
@@ -33,17 +31,15 @@ module Bootsnap
           # and the bundle path is '.bundle'.
           contains_bundle_path = BUNDLE_PATH.start_with?(path)
 
-          dirs = []
           requirables = []
           walk(path, nil) do |relative_path, absolute_path, is_directory|
             if is_directory
-              dirs << os_path(relative_path)
               !contains_bundle_path || !absolute_path.start_with?(BUNDLE_PATH)
             elsif relative_path.end_with?(*REQUIRABLE_EXTENSIONS)
-              requirables << os_path(relative_path)
+              requirables << relative_path.freeze
             end
           end
-          [requirables, dirs]
+          requirables
         end
 
         def walk(absolute_dir_path, relative_dir_path, &block)
@@ -65,15 +61,51 @@ module Bootsnap
           end
         end
 
-        if RUBY_VERSION >= "3.1"
-          def os_path(path)
-            path.freeze
+        if RUBY_ENGINE == "ruby" && RUBY_PLATFORM.match?(/darwin|linux|bsd|mswin|mingw|cygwin/)
+          require "bootsnap/bootsnap"
+        end
+
+        if defined?(Native.scan_dir)
+          def native_call(root_path)
+            # NOTE: if https://bugs.ruby-lang.org/issues/21800 is accepted we should be able
+            # to have similar performance with pure Ruby
+
+            # If the bundle path is a descendent of this path, we do additional
+            # checks to prevent recursing into the bundle path as we recurse
+            # through this path. We don't want to scan the bundle path because
+            # anything useful in it will be present on other load path items.
+            #
+            # This can happen if, for example, the user adds '.' to the load path,
+            # and the bundle path is '.bundle'.
+            contains_bundle_path = BUNDLE_PATH.start_with?(root_path)
+
+            all_requirables, queue = Native.scan_dir(root_path)
+            all_requirables.each(&:freeze)
+
+            queue.reject! do |dir|
+              ignored_directories.include?(dir) ||
+                (contains_bundle_path && dir.start_with?(BUNDLE_PATH))
+            end
+
+            while (path = queue.pop)
+              requirables, dirs = Native.scan_dir(File.join(root_path, path))
+              dirs.reject! { |dir| ignored_directories.include?(dir) }
+              dirs.map! { |f| File.join(path, f).freeze }
+              requirables.map! { |f| File.join(path, f).freeze }
+
+              if contains_bundle_path
+                dirs.reject! { |dir| dir.start_with?(BUNDLE_PATH) }
+              end
+
+              all_requirables.concat(requirables)
+              queue.concat(dirs)
+            end
+
+            all_requirables
           end
+          alias_method :call, :native_call
         else
-          def os_path(path)
-            path.force_encoding(Encoding::US_ASCII) if path.ascii_only?
-            path.freeze
-          end
+          alias_method :call, :ruby_call
         end
       end
     end

data/lib/bootsnap/load_path_cache/store.rb

@@ -8,7 +8,7 @@ module Bootsnap
   module LoadPathCache
     class Store
       VERSION_KEY = "__bootsnap_ruby_version__"
-      CURRENT_VERSION = "#{RUBY_REVISION}-#{RUBY_PLATFORM}".freeze # rubocop:disable Style/RedundantFreeze
+      CURRENT_VERSION = "#{VERSION}-#{RUBY_REVISION}-#{RUBY_PLATFORM}".freeze # rubocop:disable Style/RedundantFreeze
 
       NestedTransactionError = Class.new(StandardError)
       SetOutsideTransactionNotAllowed = Class.new(StandardError)

data/lib/bootsnap/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Bootsnap
-  VERSION = "1.18.6"
+  VERSION = "1.20.0"
 end

data/lib/bootsnap.rb

@@ -2,8 +2,8 @@
 
 require_relative "bootsnap/version"
 require_relative "bootsnap/bundler"
-require_relative "bootsnap/load_path_cache"
 require_relative "bootsnap/compile_cache"
+require_relative "bootsnap/load_path_cache"
 
 module Bootsnap
   InvalidConfiguration = Class.new(StandardError)
@@ -54,8 +54,12 @@ module Bootsnap
       revalidation: false,
       compile_cache_iseq: true,
       compile_cache_yaml: true,
-      compile_cache_json: true
+      compile_cache_json: (compile_cache_json_unset = true)
     )
+      unless compile_cache_json_unset
+        warn("Bootsnap.setup `compile_cache_json` argument is deprecated and has no effect")
+      end
+
       if load_path_cache
         Bootsnap::LoadPathCache.setup(
           cache_path: "#{cache_dir}/bootsnap/load-path-cache",
@@ -69,7 +73,6 @@ module Bootsnap
         cache_dir: "#{cache_dir}/bootsnap/compile-cache",
         iseq: compile_cache_iseq,
         yaml: compile_cache_yaml,
-        json: compile_cache_json,
         readonly: readonly,
         revalidation: revalidation,
       )
@@ -115,7 +118,6 @@ module Bootsnap
           load_path_cache: enabled?("BOOTSNAP_LOAD_PATH_CACHE"),
           compile_cache_iseq: enabled?("BOOTSNAP_COMPILE_CACHE"),
           compile_cache_yaml: enabled?("BOOTSNAP_COMPILE_CACHE"),
-          compile_cache_json: enabled?("BOOTSNAP_COMPILE_CACHE"),
           readonly: bool_env("BOOTSNAP_READONLY"),
           revalidation: bool_env("BOOTSNAP_REVALIDATE"),
           ignore_directories: ignore_directories,

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: bootsnap
 version: !ruby/object:Gem::Version
-  version: 1.18.6
+  version: 1.20.0
 platform: ruby
 authors:
 - Burke Libbey
@@ -37,7 +37,6 @@ files:
 - README.md
 - exe/bootsnap
 - ext/bootsnap/bootsnap.c
-- ext/bootsnap/bootsnap.h
 - ext/bootsnap/extconf.rb
 - lib/bootsnap.rb
 - lib/bootsnap/bundler.rb
@@ -45,7 +44,6 @@ files:
 - lib/bootsnap/cli/worker_pool.rb
 - lib/bootsnap/compile_cache.rb
 - lib/bootsnap/compile_cache/iseq.rb
-- lib/bootsnap/compile_cache/json.rb
 - lib/bootsnap/compile_cache/yaml.rb
 - lib/bootsnap/explicit_require.rb
 - lib/bootsnap/load_path_cache.rb
@@ -59,13 +57,13 @@ files:
 - lib/bootsnap/load_path_cache/store.rb
 - lib/bootsnap/setup.rb
 - lib/bootsnap/version.rb
-homepage: https://github.com/Shopify/bootsnap
+homepage: https://github.com/rails/bootsnap
 licenses:
 - MIT
 metadata:
-  bug_tracker_uri: https://github.com/Shopify/bootsnap/issues
-  changelog_uri: https://github.com/Shopify/bootsnap/blob/main/CHANGELOG.md
-  source_code_uri: https://github.com/Shopify/bootsnap
+  bug_tracker_uri: https://github.com/rails/bootsnap/issues
+  changelog_uri: https://github.com/rails/bootsnap/blob/main/CHANGELOG.md
+  source_code_uri: https://github.com/rails/bootsnap
   allowed_push_host: https://rubygems.org
 rdoc_options: []
 require_paths:
@@ -81,7 +79,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.6.8
+rubygems_version: 4.0.3
 specification_version: 4
 summary: Boot large ruby/rails apps faster
 test_files: []

data/ext/bootsnap/bootsnap.h

@@ -1,6 +0,0 @@
-#ifndef BOOTSNAP_H
-#define BOOTSNAP_H 1
-
-/* doesn't expose anything */
-
-#endif /* BOOTSNAP_H */

data/lib/bootsnap/compile_cache/json.rb

@@ -1,89 +0,0 @@
-# frozen_string_literal: true
-
-require "bootsnap/bootsnap"
-
-module Bootsnap
-  module CompileCache
-    module JSON
-      class << self
-        attr_accessor(:msgpack_factory, :supported_options)
-        attr_reader(:cache_dir)
-
-        def cache_dir=(cache_dir)
-          @cache_dir = cache_dir.end_with?("/") ? "#{cache_dir}json" : "#{cache_dir}-json"
-        end
-
-        def input_to_storage(payload, _)
-          obj = ::JSON.parse(payload)
-          msgpack_factory.dump(obj)
-        end
-
-        def storage_to_output(data, kwargs)
-          if kwargs&.key?(:symbolize_names)
-            kwargs[:symbolize_keys] = kwargs.delete(:symbolize_names)
-          end
-          msgpack_factory.load(data, kwargs)
-        end
-
-        def input_to_output(data, kwargs)
-          ::JSON.parse(data, **(kwargs || {}))
-        end
-
-        def precompile(path)
-          Bootsnap::CompileCache::Native.precompile(
-            cache_dir,
-            path.to_s,
-            self,
-          )
-        end
-
-        def install!(cache_dir)
-          self.cache_dir = cache_dir
-          init!
-          if ::JSON.respond_to?(:load_file)
-            ::JSON.singleton_class.prepend(Patch)
-          end
-        end
-
-        def init!
-          require "json"
-          require "msgpack"
-
-          self.msgpack_factory = MessagePack::Factory.new
-          self.supported_options = [:symbolize_names]
-          if supports_freeze?
-            self.supported_options = [:freeze]
-          end
-          supported_options.freeze
-        end
-
-        private
-
-        def supports_freeze?
-          ::JSON.parse('["foo"]', freeze: true).first.frozen? &&
-            MessagePack.load(MessagePack.dump("foo"), freeze: true).frozen?
-        end
-      end
-
-      module Patch
-        def load_file(path, *args)
-          return super if args.size > 1
-
-          if (kwargs = args.first)
-            return super unless kwargs.is_a?(Hash)
-            return super unless (kwargs.keys - ::Bootsnap::CompileCache::JSON.supported_options).empty?
-          end
-
-          ::Bootsnap::CompileCache::Native.fetch(
-            Bootsnap::CompileCache::JSON.cache_dir,
-            File.realpath(path),
-            ::Bootsnap::CompileCache::JSON,
-            kwargs,
-          )
-        end
-
-        ruby2_keywords :load_file if respond_to?(:ruby2_keywords, true)
-      end
-    end
-  end
-end