bootsnap 1.18.4 → 1.23.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2b2271b2fa08edc313e43ec1359be530fcb6b206a7b4b63f37065290289cbc9a
-  data.tar.gz: c5838e7d01c33320e765a2bf668c68c09fc6b0938fb8789653b526cb3c22a244
+  metadata.gz: 4e0ad269f816c24dc901b6544acb59c5ebd65b22432f39ee69bc3387b9c2b04d
+  data.tar.gz: 700f525d90d4e77421ce83e38f8731981a48cc82a8b9a937c29f661972f78d7f
 SHA512:
-  metadata.gz: 77168e700a54163bf4574fbf9f82fea681a01f7d0c5c3a55340d3dabc98959fce1176bebef4c4d25d18efa257ad5314f13ca978a77f5cb4288296c63a1b36871
-  data.tar.gz: cc770950cc3032a5033873457a23b19823c35d9d24285f3b13fb95848e51d8f34defca5396c7b545b5530381271979c93026401e4ac32d28ec0f89ceadd57823
+  metadata.gz: f5859d7fd4b81f1969c55e12b33a957b34b1dc2825d03bc3f7ab60a57d34153cbcdda7a60626ea758fd66877b1b74a58d17b276aa2055f41086488d6ced5317b
+  data.tar.gz: c09f2cd48da0ba3bb5e9d3c995ffb86b8b16aec987c311162fc79ca648a66f355e4051d6ae4c5ee486d5ad2d2bc3af32090e15776e1f45129c1ea8c8e6c811ed

data/CHANGELOG.md

@@ -1,5 +1,48 @@
 # Unreleased
 
+# 1.23.0
+
+* Require Ruby 2.7.
+* Fix support for absolute paths in `BOOTSNAP_IGNORE_DIRECTORIES`.
+
+# 1.22.0
+
+* Better fix for the `opendir` crash.
+* Add `bootsnap/rake` for cleaning the bootsnap cache as part of `rake clobber`.
+
+# 1.21.1
+
+* Prevent a Ruby crash while scanning load path if `opendir` fails without setting `errno`.
+  According to the C spec this should not happen, but according to user reports, it did.
+
+# 1.21.0
+
+* Fix the `require` decorator to handle `Bootsnap.unload_cache!` being called.
+* Minor optimization: Eagerly clear cache buffers to appease the GC.
+
+# 1.20.1
+
+* Handle broken symlinks in load path scanning code.
+  Should fix `Errno::ENOENT fstatat` issues some users have encountered after upgrading to 1.20.0.
+
+# 1.20.0
+
+* Optimized load path scanning with a C extension. Should be about 2x faster on supported platforms.
+
+# 1.19.0
+
+* Remove JSON parsing cache. Recent versions of the `json` gem are as fast as `msgpack` if not faster.
+
+# 1.18.6
+
+* Fix cgroup CPU limits detection in CLI.
+
+# 1.18.5
+
+* Attempt to detect a QEMU bug that can cause `bootsnap precompile` to hang forever when building ARM64 docker images
+  from x86_64 machines. See #495.
+* Improve CLI to detect cgroup CPU limits and avoid spawning too many worker processes.
+
 # 1.18.4
 
 * Allow using bootsnap without bundler. See #488.
@@ -95,7 +138,7 @@
 
 * Get rid of the `Kernel.require_relative` decorator by resolving `$LOAD_PATH` members to their real path.
   This way we handle symlinks in `$LOAD_PATH` much more efficiently. See #402 for the detailed explanation.
-  
+
 * Drop support for Ruby 2.3 (to allow getting rid of the `Kernel.require_relative` decorator).
 
 # 1.10.3
@@ -221,7 +264,7 @@
 * Adds an instrumentation API to monitor cache misses.
 * Allow to control the behavior of `require 'bootsnap/setup'` using environment variables.
 * Deprecate the `disable_trace` option.
-* Deprecate the `ActiveSupport::Dependencies` (AKA Classic autoloader) integration. (#344) 
+* Deprecate the `ActiveSupport::Dependencies` (AKA Classic autoloader) integration. (#344)
 
 # 1.6.0
 
@@ -241,12 +284,12 @@
 
 # 1.4.9
 
-* [Windows support](https://github.com/Shopify/bootsnap/pull/319)
-* [Fix potential crash](https://github.com/Shopify/bootsnap/pull/322)
+* [Windows support](https://github.com/rails/bootsnap/pull/319)
+* [Fix potential crash](https://github.com/rails/bootsnap/pull/322)
 
 # 1.4.8
 
-* [Prevent FallbackScan from polluting exception cause](https://github.com/Shopify/bootsnap/pull/314)
+* [Prevent FallbackScan from polluting exception cause](https://github.com/rails/bootsnap/pull/314)
 
 # 1.4.7
 
@@ -259,7 +302,7 @@
   required if a different file with the same name was already being required
 
   Example:
-  
+
       require 'foo'
       require 'foo.en'
 
@@ -313,7 +356,7 @@
 
 # 1.3.0
 
-* Handle cases where load path entries are symlinked (https://github.com/Shopify/bootsnap/pull/136)
+* Handle cases where load path entries are symlinked (https://github.com/rails/bootsnap/pull/136)
 
 # 1.2.1
 

data/LICENSE.txt

@@ -1,6 +1,7 @@
 The MIT License (MIT)
 
-Copyright (c) 2017-present Shopify, Inc.
+Copyright (c) 2017-2025 Shopify, Inc.
+Copyright (c) 2025-present Rails Foundation
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

data/README.md

@@ -1,4 +1,4 @@
-# Bootsnap [![Actions Status](https://github.com/Shopify/bootsnap/workflows/ci/badge.svg)](https://github.com/Shopify/bootsnap/actions)
+# Bootsnap [![Actions Status](https://github.com/rails/bootsnap/workflows/ci/badge.svg)](https://github.com/rails/bootsnap/actions)
 
 Bootsnap is a library that plugs into Ruby, with optional support for `YAML` and `JSON`,
 to optimize and cache expensive computations. See [How Does This Work](#how-does-this-work).
@@ -41,7 +41,7 @@ getting progressively slower, this is almost certainly the cause.**
 It's technically possible to simply specify `gem 'bootsnap', require: 'bootsnap/setup'`, but it's
 important to load Bootsnap as early as possible to get maximum performance improvement.
 
-You can see how this require works [here](https://github.com/Shopify/bootsnap/blob/main/lib/bootsnap/setup.rb).
+You can see how this require works [here](https://github.com/rails/bootsnap/blob/main/lib/bootsnap/setup.rb).
 
 If you are not using Rails, or if you are but want more control over things, add this to your
 application setup immediately after `require 'bundler/setup'` (i.e. as early as possible: the sooner
@@ -57,13 +57,12 @@ Bootsnap.setup(
   load_path_cache:      true,                 # Optimize the LOAD_PATH with a cache
   compile_cache_iseq:   true,                 # Compile Ruby code into ISeq cache, breaks coverage reporting.
   compile_cache_yaml:   true,                 # Compile YAML into a cache
-  compile_cache_json:   true,                 # Compile JSON into a cache
   readonly:             true,                 # Use the caches but don't update them on miss or stale entries.
 )
 ```
 
 **Protip:** You can replace `require 'bootsnap'` with `BootLib::Require.from_gem('bootsnap',
-'bootsnap')` using [this trick](https://github.com/Shopify/bootsnap/wiki/Bootlib::Require). This
+'bootsnap')` using [this trick](https://github.com/rails/bootsnap/wiki/Bootlib::Require). This
 will help optimize boot time further if you have an extremely large `$LOAD_PATH`.
 
 Note: Bootsnap and [Spring](https://github.com/rails/spring) are orthogonal tools. While Bootsnap
@@ -170,7 +169,7 @@ The only directories considered "stable" are things under the Ruby install prefi
 "volatile".
 
 In addition to the [`Bootsnap::LoadPathCache::Cache`
-source](https://github.com/Shopify/bootsnap/blob/main/lib/bootsnap/load_path_cache/cache.rb),
+source](https://github.com/rails/bootsnap/blob/main/lib/bootsnap/load_path_cache/cache.rb),
 this diagram may help clarify how entry resolution works:
 
 ![How path searching works](https://cloud.githubusercontent.com/assets/3074765/25388270/670b5652-299b-11e7-87fb-975647f68981.png)
@@ -188,7 +187,7 @@ result too, raising a `LoadError` without touching the filesystem at all.
 
 Ruby has complex grammar and parsing it is not a particularly cheap operation. Since 1.9, Ruby has
 translated ruby source to an internal bytecode format, which is then executed by the Ruby VM. Since
-2.3.0, Ruby [exposes an API](https://ruby-doc.org/core-2.3.0/RubyVM/InstructionSequence.html) that
+2.3.0, Ruby [exposes an API](https://docs.ruby-lang.org/en/master/RubyVM/InstructionSequence.html) that
 allows caching that bytecode. This allows us to bypass the relatively-expensive compilation step on
 subsequent loads of the same file.
 
@@ -331,9 +330,23 @@ To do so you can use the `bootsnap precompile` command.
 Example:
 
 ```bash
-$ bundle exec bootsnap precompile --gemfile app/ lib/
+$ bundle exec bootsnap precompile --gemfile app/ lib/ config/
 ```
 
+## Known issues
+
+### QEMU environments
+
+When building cross-platform Docker images, QEMU is often used for emulation and can be the source of a limitation that causes forked processes to hang. While Bootsnap includes automatic detection for this issue (as of [PR #501](https://github.com/rails/bootsnap/pull/501)), the detection may not always be sufficient.
+
+If you encounter hangs during precompilation in QEMU-based environments (such as when using Docker buildx for cross-platform builds), you can work around this by disabling parallelization with the `-j 0` option:
+
+```bash
+$ bundle exec bootsnap precompile -j 0 --gemfile app/ lib/ config/
+```
+
+See [Issue #495](https://github.com/rails/bootsnap/issues/495) for more details about this QEMU-related issue.
+
 ## When not to use Bootsnap
 
 *Alternative engines*: Bootsnap is pretty reliant on MRI features, and parts are disabled entirely on alternative ruby

data/ext/bootsnap/bootsnap.c

@@ -11,7 +11,6 @@
  * here.
  */
 
-#include "bootsnap.h"
 #include "ruby.h"
 #include <stdint.h>
 #include <stdbool.h>
@@ -20,10 +19,15 @@
 #include <fcntl.h>
 #include <unistd.h>
 #include <sys/stat.h>
+#include <dirent.h>
+
+#ifndef RBIMPL_ATTR_NORETURN
+#define RBIMPL_ATTR_NORETURN()
+#endif
 
 #ifdef __APPLE__
   // The symbol is present, however not in the headers
-  // See: https://github.com/Shopify/bootsnap/issues/470
+  // See: https://github.com/rails/bootsnap/issues/470
   extern int fdatasync(int);
 #endif
 
@@ -96,7 +100,6 @@ static mode_t current_umask;
 
 /* Bootsnap::CompileCache::{Native, Uncompilable} */
 static VALUE rb_mBootsnap;
-static VALUE rb_mBootsnap_CompileCache;
 static VALUE rb_mBootsnap_CompileCache_Native;
 static VALUE rb_cBootsnap_CompileCache_UNCOMPILABLE;
 static ID instrumentation_method;
@@ -152,6 +155,114 @@ bs_rb_get_path(VALUE self, VALUE fname)
     return rb_get_path(fname);
 }
 
+#ifdef HAVE_FSTATAT
+
+RBIMPL_ATTR_NORETURN()
+static void
+bs_syserr_fail_path(const char *func_name, int n, VALUE path)
+{
+    rb_syserr_fail_str(n, rb_sprintf("%s @ %s", func_name, RSTRING_PTR(path)));
+}
+
+RBIMPL_ATTR_NORETURN()
+static void
+bs_syserr_fail_dir_entry(const char *func_name, int n, VALUE dir, const char *d_name)
+{
+    rb_syserr_fail_str(n, rb_sprintf("%s @ %s/%s", func_name, RSTRING_PTR(dir), d_name));
+}
+
+static VALUE
+bs_rb_scan_dir(VALUE self, VALUE abspath)
+{
+    Check_Type(abspath, T_STRING);
+
+    VALUE dirs = rb_ary_new();
+    VALUE requirables = rb_ary_new();
+    VALUE result = rb_ary_new_from_args(2, requirables, dirs);
+
+    DIR *dirp = opendir(RSTRING_PTR(abspath));
+    if (dirp == NULL) {
+        if (errno == ENOTDIR || errno == ENOENT) {
+            return result;
+        }
+
+        bs_syserr_fail_path("opendir", errno, abspath);
+        return Qundef;
+    }
+
+    struct dirent *entry;
+    struct stat st;
+    int dfd = -1;
+
+    while (1) {
+        errno = 0;
+
+        entry = readdir(dirp);
+        if (entry == NULL) break;
+
+        if (entry->d_name[0] == '.') continue;
+
+        if (RB_UNLIKELY(entry->d_type == DT_UNKNOWN || entry->d_type == DT_LNK)) {
+            // Note: the original implementation of LoadPathCache did follow symlink.
+            // So this is replicated here, but I'm not sure it's a good idea.
+            if (dfd < 0) {
+                dfd = dirfd(dirp);
+                if (dfd < 0) {
+                    int err = errno;
+                    closedir(dirp);
+                    bs_syserr_fail_path("dirfd", err, abspath);
+                    return Qundef;
+                }
+            }
+
+            if (fstatat(dfd, entry->d_name, &st, 0)) {
+                if (errno == ENOENT) continue; // Broken symlink
+
+                int err = errno;
+                closedir(dirp);
+                bs_syserr_fail_dir_entry("fstatat", err, abspath, entry->d_name);
+                return Qundef;
+            }
+
+            if (S_ISREG(st.st_mode)) {
+                entry->d_type = DT_REG;
+            } else if (S_ISDIR(st.st_mode)) {
+                entry->d_type = DT_DIR;
+            }
+        }
+
+        if (entry->d_type == DT_DIR) {
+            rb_ary_push(dirs, rb_utf8_str_new_cstr(entry->d_name));
+            continue;
+        } else if (entry->d_type == DT_REG) {
+            size_t len = strlen(entry->d_name);
+            bool is_requirable = (
+                // Comparing 4B allows compiler to optimize this into a single 32b integer comparison.
+                (len > 3 && memcmp(entry->d_name + (len - 3), ".rb", 4) == 0)
+                || (len > DLEXT_MAXLEN && memcmp(entry->d_name + (len - DLEXT_MAXLEN), DLEXT, DLEXT_MAXLEN + 1) == 0)
+#ifdef DLEXT2
+                || (len > DLEXT2_MAXLEN && memcmp(entry->d_name + (len - DLEXT2_MAXLEN), DLEXT2, DLEXT2_MAXLEN + 1) == 0)
+#endif
+            );
+            if (is_requirable) {
+                rb_ary_push(requirables, rb_utf8_str_new(entry->d_name, len));
+            }
+        }
+    }
+
+    if (errno) {
+      int err = errno;
+      closedir(dirp);
+      bs_syserr_fail_path("readdir", err, abspath);
+    } else if (closedir(dirp)) {
+        bs_syserr_fail_path("closedir", errno, abspath);
+        return Qundef;
+    }
+
+    return result;
+}
+#endif
+
 /*
  * Ruby C extensions are initialized by calling Init_<extname>.
  *
@@ -166,7 +277,14 @@ Init_bootsnap(void)
 
   rb_define_singleton_method(rb_mBootsnap, "rb_get_path", bs_rb_get_path, 1);
 
-  rb_mBootsnap_CompileCache = rb_define_module_under(rb_mBootsnap, "CompileCache");
+#ifdef HAVE_FSTATAT
+  VALUE rb_mBootsnap_LoadPathCache = rb_define_module_under(rb_mBootsnap, "LoadPathCache");
+  VALUE rb_mBootsnap_LoadPathCache_Native = rb_define_module_under(rb_mBootsnap_LoadPathCache, "Native");
+
+  rb_define_singleton_method(rb_mBootsnap_LoadPathCache_Native, "scan_dir", bs_rb_scan_dir, 1);
+#endif
+
+  VALUE rb_mBootsnap_CompileCache = rb_define_module_under(rb_mBootsnap, "CompileCache");
   rb_mBootsnap_CompileCache_Native = rb_define_module_under(rb_mBootsnap_CompileCache, "Native");
   rb_cBootsnap_CompileCache_UNCOMPILABLE = rb_const_get(rb_mBootsnap_CompileCache, rb_intern("UNCOMPILABLE"));
   rb_global_variable(&rb_cBootsnap_CompileCache_UNCOMPILABLE);

data/ext/bootsnap/extconf.rb

@@ -4,6 +4,7 @@ require "mkmf"
 
 if %w[ruby truffleruby].include?(RUBY_ENGINE)
   have_func "fdatasync", "unistd.h"
+  have_func "fstatat", "sys/stat.h"
 
   unless RUBY_PLATFORM.match?(/mswin|mingw|cygwin/)
     append_cppflags ["-D_GNU_SOURCE"] # Needed of O_NOATIME
@@ -12,7 +13,7 @@ if %w[ruby truffleruby].include?(RUBY_ENGINE)
   append_cflags ["-O3", "-std=c99"]
 
   # ruby.h has some -Wpedantic fails in some cases
-  # (e.g. https://github.com/Shopify/bootsnap/issues/15)
+  # (e.g. https://github.com/rails/bootsnap/issues/15)
   unless ["0", "", nil].include?(ENV["BOOTSNAP_PEDANTIC"])
     append_cflags([
       "-Wall",

data/lib/bootsnap/cli/worker_pool.rb

@@ -1,16 +1,88 @@
 # frozen_string_literal: true
 
+require "etc"
+require "rbconfig"
+require "io/wait" unless IO.method_defined?(:wait_readable)
+
 module Bootsnap
   class CLI
     class WorkerPool
       class << self
         def create(size:, jobs:)
+          size ||= default_size
           if size > 0 && Process.respond_to?(:fork)
             new(size: size, jobs: jobs)
           else
             Inline.new(jobs: jobs)
           end
         end
+
+        def default_size
+          nprocessors = Etc.nprocessors
+          size = [nprocessors, cpu_quota&.to_i || nprocessors].min
+          case size
+          when 0, 1
+            0
+          else
+            if fork_defunct?
+              $stderr.puts "warning: faulty fork(2) detected, probably in cross platform docker builds. " \
+                           "Disabling parallel compilation."
+              0
+            else
+              size
+            end
+          end
+        end
+
+        def cpu_quota
+          if RbConfig::CONFIG["target_os"].include?("linux")
+            if File.exist?("/sys/fs/cgroup/cpu.max")
+              # cgroups v2: https://docs.kernel.org/admin-guide/cgroup-v2.html#cpu-interface-files
+              cpu_max = File.read("/sys/fs/cgroup/cpu.max")
+              return nil if cpu_max.start_with?("max ") # no limit
+
+              max, period = cpu_max.split.map(&:to_f)
+              max / period
+            elsif File.exist?("/sys/fs/cgroup/cpu,cpuacct/cpu.cfs_quota_us")
+              # cgroups v1: https://kernel.googlesource.com/pub/scm/linux/kernel/git/glommer/memcg/+/cpu_stat/Documentation/cgroups/cpu.txt
+              max = File.read("/sys/fs/cgroup/cpu,cpuacct/cpu.cfs_quota_us").to_i
+              # If the cpu.cfs_quota_us is -1, cgroup does not adhere to any CPU time restrictions
+              # https://docs.kernel.org/scheduler/sched-bwc.html#management
+              return nil if max <= 0
+
+              period = File.read("/sys/fs/cgroup/cpu,cpuacct/cpu.cfs_period_us").to_f
+              max / period
+            end
+          end
+        end
+
+        def fork_defunct?
+          return true unless ::Process.respond_to?(:fork)
+
+          # Ref: https://github.com/rails/bootsnap/issues/495
+          # The second forked process will hang on some QEMU environments
+          r, w = IO.pipe
+          pids = 2.times.map do
+            ::Process.fork do
+              exit!(true)
+            end
+          end
+          w.close
+          r.wait_readable(1) # Wait at most 1s
+
+          defunct = false
+
+          pids.each do |pid|
+            _pid, status = ::Process.wait2(pid, ::Process::WNOHANG)
+            if status.nil? # Didn't exit in 1s
+              defunct = true
+              Process.kill(:KILL, pid)
+              ::Process.wait2(pid)
+            end
+          end
+
+          defunct
+        end
       end
 
       class Inline

data/lib/bootsnap/cli.rb

@@ -4,7 +4,6 @@ require "bootsnap"
 require "bootsnap/cli/worker_pool"
 require "optparse"
 require "fileutils"
-require "etc"
 
 module Bootsnap
   class CLI
@@ -21,7 +20,7 @@ module Bootsnap
 
     attr_reader :cache_dir, :argv
 
-    attr_accessor :compile_gemfile, :exclude, :verbose, :iseq, :yaml, :json, :jobs
+    attr_accessor :compile_gemfile, :exclude, :verbose, :iseq, :yaml, :jobs
 
     def initialize(argv)
       @argv = argv
@@ -29,10 +28,9 @@ module Bootsnap
       self.compile_gemfile = false
       self.exclude = nil
       self.verbose = false
-      self.jobs = Etc.nprocessors
+      self.jobs = nil
       self.iseq = true
       self.yaml = true
-      self.json = true
     end
 
     def precompile_command(*sources)
@@ -43,21 +41,18 @@ module Bootsnap
           cache_dir: cache_dir,
           iseq: iseq,
           yaml: yaml,
-          json: json,
           revalidation: true,
         )
 
         @work_pool = WorkerPool.create(size: jobs, jobs: {
           ruby: method(:precompile_ruby),
           yaml: method(:precompile_yaml),
-          json: method(:precompile_json),
         })
         @work_pool.spawn
 
         main_sources = sources.map { |d| File.expand_path(d) }
         precompile_ruby_files(main_sources)
         precompile_yaml_files(main_sources)
-        precompile_json_files(main_sources)
 
         if compile_gemfile
           # Gems that include JSON or YAML files usually don't put them in `lib/`.
@@ -71,7 +66,6 @@ module Bootsnap
 
           precompile_ruby_files(gem_paths, exclude: gem_exclude)
           precompile_yaml_files(gem_paths, exclude: gem_exclude)
-          precompile_json_files(gem_paths, exclude: gem_exclude)
         end
 
         if (exitstatus = @work_pool.shutdown)
@@ -146,29 +140,6 @@ module Bootsnap
       end
     end
 
-    def precompile_json_files(load_paths, exclude: self.exclude)
-      return unless json
-
-      load_paths.each do |path|
-        if !exclude || !exclude.match?(path)
-          list_files(path, "**/*.json").each do |json_file|
-            # We ignore hidden files to not match the various .config.json files
-            if !File.basename(json_file).start_with?(".") && (!exclude || !exclude.match?(json_file))
-              @work_pool.push(:json, json_file)
-            end
-          end
-        end
-      end
-    end
-
-    def precompile_json(*json_files)
-      Array(json_files).each do |json_file|
-        if CompileCache::JSON.precompile(json_file) && verbose
-          $stderr.puts(json_file)
-        end
-      end
-    end
-
     def precompile_ruby_files(load_paths, exclude: self.exclude)
       return unless iseq
 
@@ -277,9 +248,9 @@ module Bootsnap
         opts.on("--no-yaml", help) { self.yaml = false }
 
         help = <<~HELP
-          Disable JSON precompilation.
+          Disable JSON precompilation. Deprecated.
         HELP
-        opts.on("--no-json", help) { self.json = false }
+        opts.on("--no-json", help) { $stderr.puts("The --no-json option is deprecated and now a noop.") }
       end
     end
   end

data/lib/bootsnap/compile_cache/iseq.rb

@@ -50,7 +50,9 @@ module Bootsnap
       end
 
       def self.storage_to_output(binary, _args)
-        RubyVM::InstructionSequence.load_from_binary(binary)
+        iseq = RubyVM::InstructionSequence.load_from_binary(binary)
+        binary.clear
+        iseq
       rescue RuntimeError => error
         if error.message == "broken binary format"
           $stderr.puts("[Bootsnap::CompileCache] warning: rejecting broken binary")
@@ -95,7 +97,7 @@ module Bootsnap
         end
 
         def compile_option=(hash)
-          super(hash)
+          super
           Bootsnap::CompileCache::ISeq.compile_option_updated
         end
       end

data/lib/bootsnap/compile_cache/yaml.rb

@@ -170,7 +170,9 @@ module Bootsnap
             unpacker = CompileCache::YAML.msgpack_factory.unpacker(kwargs)
             unpacker.feed(data)
             _safe_loaded = unpacker.unpack
-            unpacker.unpack
+            result = unpacker.unpack
+            data.clear
+            result
           end
 
           def input_to_output(data, kwargs)

data/lib/bootsnap/compile_cache.rb

@@ -9,7 +9,11 @@ module Bootsnap
 
     Error = Class.new(StandardError)
 
-    def self.setup(cache_dir:, iseq:, yaml:, json:, readonly: false, revalidation: false)
+    def self.setup(cache_dir:, iseq:, yaml:, json: (json_unset = true), readonly: false, revalidation: false)
+      unless json_unset
+        warn("Bootsnap::CompileCache.setup `json` argument is deprecated and has no effect")
+      end
+
       if iseq
         if supported?
           require_relative "compile_cache/iseq"
@@ -28,15 +32,6 @@ module Bootsnap
         end
       end
 
-      if json
-        if supported?
-          require_relative "compile_cache/json"
-          Bootsnap::CompileCache::JSON.install!(cache_dir)
-        elsif $VERBOSE
-          warn("[bootsnap/setup] JSON parsing caching is not supported on this implementation of Ruby")
-        end
-      end
-
       if supported? && defined?(Bootsnap::CompileCache::Native)
         Bootsnap::CompileCache::Native.readonly = readonly
         Bootsnap::CompileCache::Native.revalidation = revalidation

data/lib/bootsnap/load_path_cache/cache.rb

@@ -11,19 +11,19 @@ module Bootsnap
         @development_mode = development_mode
         @store = store
         @mutex = Mutex.new
-        @path_obj = path_obj.map! { |f| PathScanner.os_path(File.exist?(f) ? File.realpath(f) : f.dup) }
+        @path_obj = path_obj.map! do |f|
+          if File.exist?(f)
+            File.realpath(f).freeze
+          elsif f.frozen?
+            f
+          else
+            f.dup.freeze
+          end
+        end
         @has_relative_paths = nil
         reinitialize
       end
 
-      # What is the path item that contains the dir as child?
-      # e.g. given "/a/b/c/d" exists, and the path is ["/a/b"], load_dir("c/d")
-      # is "/a/b".
-      def load_dir(dir)
-        reinitialize if stale?
-        @mutex.synchronize { @dirs[dir] }
-      end
-
       TRUFFLERUBY_LIB_DIR_PREFIX = if RUBY_ENGINE == "truffleruby"
         "#{File.join(RbConfig::CONFIG['libdir'], 'truffle')}#{File::SEPARATOR}"
       end
@@ -124,7 +124,6 @@ module Bootsnap
           @path_obj = path_obj
           ChangeObserver.register(@path_obj, self)
           @index = {}
-          @dirs = {}
           @generated_at = now
           push_paths_locked(*@path_obj)
         end
@@ -152,9 +151,8 @@ module Bootsnap
             p = p.to_realpath
 
             expanded_path = p.expanded_path
-            entries, dirs = p.entries_and_dirs(@store)
+            entries = p.entries(@store)
             # push -> low precedence -> set only if unset
-            dirs.each    { |dir| @dirs[dir] ||= path }
             entries.each { |rel| @index[rel] ||= expanded_path }
           end
         end
@@ -169,9 +167,8 @@ module Bootsnap
             p = p.to_realpath
 
             expanded_path = p.expanded_path
-            entries, dirs = p.entries_and_dirs(@store)
+            entries = p.entries(@store)
             # unshift -> high precedence -> unconditional set
-            dirs.each    { |dir| @dirs[dir]  = path }
             entries.each { |rel| @index[rel] = expanded_path }
           end
         end

data/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb

@@ -5,7 +5,7 @@ module Kernel
 
   alias_method :require, :require # Avoid method redefinition warnings
 
-  def require(path) # rubocop:disable Lint/DuplicateMethods
+  def require(path)
     return require_without_bootsnap(path) unless Bootsnap::LoadPathCache.enabled?
 
     string_path = Bootsnap.rb_get_path(path)
@@ -15,8 +15,11 @@ module Kernel
     if Bootsnap::LoadPathCache::FALLBACK_SCAN.equal?(resolved)
       if (cursor = Bootsnap::LoadPathCache.loaded_features_index.cursor(string_path))
         ret = require_without_bootsnap(path)
-        resolved = Bootsnap::LoadPathCache.loaded_features_index.identify(string_path, cursor)
-        Bootsnap::LoadPathCache.loaded_features_index.register(string_path, resolved)
+
+        # The file we required may have unloaded the cache
+        resolved = Bootsnap::LoadPathCache.loaded_features_index&.identify(string_path, cursor)
+        Bootsnap::LoadPathCache.loaded_features_index&.register(string_path, resolved)
+
         return ret
       else
         return require_without_bootsnap(path)
@@ -28,7 +31,9 @@ module Kernel
     else
       # Note that require registers to $LOADED_FEATURES while load does not.
       ret = require_without_bootsnap(resolved)
-      Bootsnap::LoadPathCache.loaded_features_index.register(string_path, resolved)
+
+      # The file we required may have unloaded the cache
+      Bootsnap::LoadPathCache.loaded_features_index&.register(string_path, resolved)
       return ret
     end
   end

data/lib/bootsnap/load_path_cache/loaded_features_index.rb

@@ -142,7 +142,7 @@ module Bootsnap
       # will _never_ run on MacOS, and therefore think they can get away
       # with calling a Ruby file 'x.dylib.rb' and then requiring it as 'x.dylib'.)
       #
-      # See <https://ruby-doc.org/core-2.6.4/Kernel.html#method-i-require>.
+      # See <https://docs.ruby-lang.org/en/master/Kernel.html#method-i-require>.
       def extension_elidable?(feature)
         feature.to_s.end_with?(".rb", ".so", ".o", ".dll", ".dylib")
       end

data/lib/bootsnap/load_path_cache/path.rb

@@ -54,29 +54,28 @@ module Bootsnap
         !path.start_with?(SLASH)
       end
 
-      # Return a list of all the requirable files and all of the subdirectories
-      # of this +Path+.
-      def entries_and_dirs(store)
+      # Return a list of all the requirable files of this +Path+.
+      def entries(store)
         if stable?
           # the cached_mtime field is unused for 'stable' paths, but is
           # set to zero anyway, just in case we change the stability heuristics.
-          _, entries, dirs = store.get(expanded_path)
-          return [entries, dirs] if entries # cache hit
+          _, entries, = store.get(expanded_path)
+          return entries if entries # cache hit
 
-          entries, dirs = scan!
-          store.set(expanded_path, [0, entries, dirs])
-          return [entries, dirs]
+          entries = PathScanner.call(expanded_path)
+          store.set(expanded_path, [0, entries])
+          return entries
         end
 
-        cached_mtime, entries, dirs = store.get(expanded_path)
+        cached_mtime, entries = store.get(expanded_path)
 
-        current_mtime = latest_mtime(expanded_path, dirs || [])
-        return [[], []]        if current_mtime == -1 # path does not exist
-        return [entries, dirs] if cached_mtime == current_mtime
+        current_mtime = latest_mtime(expanded_path, entries || [])
+        return [] if current_mtime == -1 # path does not exist
+        return entries if cached_mtime == current_mtime
 
-        entries, dirs = scan!
-        store.set(expanded_path, [current_mtime, entries, dirs])
-        [entries, dirs]
+        entries = PathScanner.call(expanded_path)
+        store.set(expanded_path, [current_mtime, entries])
+        entries
       end
 
       def expanded_path
@@ -89,23 +88,31 @@ module Bootsnap
 
       private
 
-      def scan! # (expensive) returns [entries, dirs]
-        PathScanner.call(expanded_path)
-      end
-
       # last time a directory was modified in this subtree. +dirs+ should be a
       # list of relative paths to directories under +path+. e.g. for /a/b and
       # /a/b/c, pass ('/a/b', ['c'])
-      def latest_mtime(path, dirs)
-        max = -1
-        ["", *dirs].each do |dir|
-          curr = begin
-            File.mtime("#{path}/#{dir}").to_i
-                 rescue Errno::ENOENT, Errno::ENOTDIR, Errno::EINVAL
-                   -1
+      def latest_mtime(root_path, entries)
+        max = begin
+          File.mtime(root_path).to_i
+        rescue Errno::ENOENT, Errno::ENOTDIR, Errno::EINVAL
+          -1
+        end
+
+        visited = {"." => true}
+
+        entries.each do |relpath|
+          dirname = File.dirname(relpath)
+          visited[dirname] ||= begin
+            begin
+              current = File.mtime(File.join(root_path, dirname)).to_i
+              max = current if current > max
+            rescue Errno::ENOENT, Errno::ENOTDIR, Errno::EINVAL
+              # ignore
+            end
+            true
           end
-          max = curr if curr > max
         end
+
         max
       end
 

data/lib/bootsnap/load_path_cache/path_scanner.rb

@@ -6,8 +6,6 @@ module Bootsnap
   module LoadPathCache
     module PathScanner
       REQUIRABLE_EXTENSIONS = [DOT_RB] + DL_EXTENSIONS
-      NORMALIZE_NATIVE_EXTENSIONS = !DL_EXTENSIONS.include?(LoadPathCache::DOT_SO)
-      ALTERNATIVE_NATIVE_EXTENSIONS_PATTERN = /\.(o|bundle|dylib)\z/.freeze
 
       BUNDLE_PATH = if Bootsnap.bundler?
         (Bundler.bundle_path.cleanpath.to_s << LoadPathCache::SLASH).freeze
@@ -20,9 +18,74 @@ module Bootsnap
       class << self
         attr_accessor :ignored_directories
 
-        def call(path)
-          path = File.expand_path(path.to_s).freeze
-          return [[], []] unless File.directory?(path)
+        def ruby_call(root_path)
+          root_path, contains_bundle_path, ignored_abs_paths, ignored_dir_names = prepare_scan(root_path)
+          return [] unless File.directory?(root_path)
+
+          requirables = []
+          walk(root_path, nil, ignored_abs_paths, ignored_dir_names) do |relative_path, absolute_path, is_directory|
+            if is_directory
+              !contains_bundle_path || !absolute_path.start_with?(BUNDLE_PATH)
+            elsif relative_path.end_with?(*REQUIRABLE_EXTENSIONS)
+              requirables << relative_path.freeze
+            end
+          end
+          requirables
+        end
+
+        if RUBY_ENGINE == "ruby" && RUBY_PLATFORM.match?(/darwin|linux|bsd|mswin|mingw|cygwin/)
+          require "bootsnap/bootsnap"
+        end
+
+        if defined?(Native.scan_dir)
+          def native_call(root_path)
+            # NOTE: if https://bugs.ruby-lang.org/issues/21800 is accepted we should be able
+            # to have similar performance with pure Ruby
+            root_path, contains_bundle_path, ignored_abs_paths, ignored_dir_names = prepare_scan(root_path)
+
+            all_requirables, queue = Native.scan_dir(root_path)
+            all_requirables.each(&:freeze)
+
+            queue.reject! do |dir|
+              if ignored_dir_names&.include?(dir)
+                true
+              elsif ignored_abs_paths || contains_bundle_path
+                absolute_dir = File.join(root_path, dir)
+                ignored_abs_paths&.include?(absolute_dir) ||
+                  (contains_bundle_path && absolute_dir.start_with?(BUNDLE_PATH))
+              end
+            end
+
+            while (relative_path = queue.pop)
+              absolute_base = File.join(root_path, relative_path)
+              requirables, dirs = Native.scan_dir(absolute_base)
+              dirs.reject! do |dir|
+                if ignored_dir_names&.include?(dir)
+                  true
+                elsif ignored_abs_paths || contains_bundle_path
+                  absolute_dir = File.join(absolute_base, dir)
+                  ignored_abs_paths&.include?(absolute_dir) ||
+                    (contains_bundle_path && absolute_dir.start_with?(BUNDLE_PATH))
+                end
+              end
+              dirs.map! { |f| File.join(relative_path, f).freeze }
+              requirables.map! { |f| File.join(relative_path, f).freeze }
+
+              all_requirables.concat(requirables)
+              queue.concat(dirs)
+            end
+
+            all_requirables
+          end
+          alias_method :call, :native_call
+        else
+          alias_method :call, :ruby_call
+        end
+
+        private
+
+        def prepare_scan(root_path)
+          root_path = File.expand_path(root_path.to_s).freeze
 
           # If the bundle path is a descendent of this path, we do additional
           # checks to prevent recursing into the bundle path as we recurse
@@ -31,50 +94,33 @@ module Bootsnap
           #
           # This can happen if, for example, the user adds '.' to the load path,
           # and the bundle path is '.bundle'.
-          contains_bundle_path = BUNDLE_PATH.start_with?(path)
+          contains_bundle_path = BUNDLE_PATH.start_with?(root_path)
 
-          dirs = []
-          requirables = []
-          walk(path, nil) do |relative_path, absolute_path, is_directory|
-            if is_directory
-              dirs << os_path(relative_path)
-              !contains_bundle_path || !absolute_path.start_with?(BUNDLE_PATH)
-            elsif relative_path.end_with?(*REQUIRABLE_EXTENSIONS)
-              requirables << os_path(relative_path)
-            end
-          end
-          [requirables, dirs]
+          ignored_abs_paths, ignored_dir_names = ignored_directories.partition { |p| File.absolute_path?(p) }
+          ignored_abs_paths = nil if ignored_abs_paths.empty?
+          ignored_dir_names = nil if ignored_dir_names.empty?
+
+          [root_path, contains_bundle_path, ignored_abs_paths, ignored_dir_names]
         end
 
-        def walk(absolute_dir_path, relative_dir_path, &block)
+        def walk(absolute_dir_path, relative_dir_path, ignored_abs_paths, ignored_dir_names, &block)
           Dir.foreach(absolute_dir_path) do |name|
             next if name.start_with?(".")
 
             relative_path = relative_dir_path ? File.join(relative_dir_path, name) : name
 
-            absolute_path = "#{absolute_dir_path}/#{name}"
+            absolute_path = File.join(absolute_dir_path, name)
             if File.directory?(absolute_path)
-              next if ignored_directories.include?(name) || ignored_directories.include?(absolute_path)
+              next if ignored_dir_names&.include?(name) || ignored_abs_paths&.include?(absolute_path)
 
               if yield relative_path, absolute_path, true
-                walk(absolute_path, relative_path, &block)
+                walk(absolute_path, relative_path, ignored_abs_paths, ignored_dir_names, &block)
               end
             else
               yield relative_path, absolute_path, false
             end
           end
         end
-
-        if RUBY_VERSION >= "3.1"
-          def os_path(path)
-            path.freeze
-          end
-        else
-          def os_path(path)
-            path.force_encoding(Encoding::US_ASCII) if path.ascii_only?
-            path.freeze
-          end
-        end
       end
     end
   end

data/lib/bootsnap/load_path_cache/store.rb

@@ -8,7 +8,7 @@ module Bootsnap
   module LoadPathCache
     class Store
       VERSION_KEY = "__bootsnap_ruby_version__"
-      CURRENT_VERSION = "#{RUBY_REVISION}-#{RUBY_PLATFORM}".freeze # rubocop:disable Style/RedundantFreeze
+      CURRENT_VERSION = "#{VERSION}-#{RUBY_REVISION}-#{RUBY_PLATFORM}".freeze # rubocop:disable Style/RedundantFreeze
 
       NestedTransactionError = Class.new(StandardError)
       SetOutsideTransactionNotAllowed = Class.new(StandardError)

data/lib/bootsnap/rake.rb

@@ -0,0 +1,14 @@
+# frozen_string_literal: true
+
+require "bootsnap"
+require "rake/clean"
+
+# Typically, should have been set up prior to requiring rake integration.
+# But allow for a streamlined ::default_setup
+require "bootsnap/setup" if Bootsnap.cache_dir.nil?
+
+if Bootsnap.cache_dir
+  CLEAN.include Bootsnap.cache_dir
+else
+  abort "Bootsnap must be set-up prior to rake integration"
+end

data/lib/bootsnap/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Bootsnap
-  VERSION = "1.18.4"
+  VERSION = "1.23.0"
 end

data/lib/bootsnap.rb

@@ -2,14 +2,12 @@
 
 require_relative "bootsnap/version"
 require_relative "bootsnap/bundler"
-require_relative "bootsnap/load_path_cache"
-require_relative "bootsnap/compile_cache"
 
 module Bootsnap
   InvalidConfiguration = Class.new(StandardError)
 
   class << self
-    attr_reader :logger
+    attr_reader :cache_dir, :logger
 
     def log_stats!
       stats = {hit: 0, revalidated: 0, miss: 0, stale: 0}
@@ -54,11 +52,17 @@ module Bootsnap
       revalidation: false,
       compile_cache_iseq: true,
       compile_cache_yaml: true,
-      compile_cache_json: true
+      compile_cache_json: (compile_cache_json_unset = true)
     )
+      unless compile_cache_json_unset
+        warn("Bootsnap.setup `compile_cache_json` argument is deprecated and has no effect")
+      end
+
+      @cache_dir = "#{cache_dir}/bootsnap"
+
       if load_path_cache
         Bootsnap::LoadPathCache.setup(
-          cache_path: "#{cache_dir}/bootsnap/load-path-cache",
+          cache_path: "#{@cache_dir}/load-path-cache",
           development_mode: development_mode,
           ignore_directories: ignore_directories,
           readonly: readonly,
@@ -66,10 +70,9 @@ module Bootsnap
       end
 
       Bootsnap::CompileCache.setup(
-        cache_dir: "#{cache_dir}/bootsnap/compile-cache",
+        cache_dir: "#{@cache_dir}/compile-cache",
         iseq: compile_cache_iseq,
         yaml: compile_cache_yaml,
-        json: compile_cache_json,
         readonly: readonly,
         revalidation: revalidation,
       )
@@ -115,7 +118,6 @@ module Bootsnap
           load_path_cache: enabled?("BOOTSNAP_LOAD_PATH_CACHE"),
           compile_cache_iseq: enabled?("BOOTSNAP_COMPILE_CACHE"),
           compile_cache_yaml: enabled?("BOOTSNAP_COMPILE_CACHE"),
-          compile_cache_json: enabled?("BOOTSNAP_COMPILE_CACHE"),
           readonly: bool_env("BOOTSNAP_READONLY"),
           revalidation: bool_env("BOOTSNAP_REVALIDATE"),
           ignore_directories: ignore_directories,
@@ -148,7 +150,7 @@ module Bootsnap
     end
 
     # Allow the C extension to redefine `rb_get_path` without warning.
-    alias_method :rb_get_path, :rb_get_path # rubocop:disable Lint/DuplicateMethods
+    alias_method :rb_get_path, :rb_get_path
 
     private
 
@@ -162,3 +164,6 @@ module Bootsnap
     end
   end
 end
+
+require_relative "bootsnap/compile_cache"
+require_relative "bootsnap/load_path_cache"

metadata

@@ -1,14 +1,13 @@
 --- !ruby/object:Gem::Specification
 name: bootsnap
 version: !ruby/object:Gem::Version
-  version: 1.18.4
+  version: 1.23.0
 platform: ruby
 authors:
 - Burke Libbey
-autorequire:
 bindir: exe
 cert_chain: []
-date: 2024-08-05 00:00:00.000000000 Z
+date: 1980-01-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: msgpack
@@ -38,7 +37,6 @@ files:
 - README.md
 - exe/bootsnap
 - ext/bootsnap/bootsnap.c
-- ext/bootsnap/bootsnap.h
 - ext/bootsnap/extconf.rb
 - lib/bootsnap.rb
 - lib/bootsnap/bundler.rb
@@ -46,7 +44,6 @@ files:
 - lib/bootsnap/cli/worker_pool.rb
 - lib/bootsnap/compile_cache.rb
 - lib/bootsnap/compile_cache/iseq.rb
-- lib/bootsnap/compile_cache/json.rb
 - lib/bootsnap/compile_cache/yaml.rb
 - lib/bootsnap/explicit_require.rb
 - lib/bootsnap/load_path_cache.rb
@@ -58,17 +55,17 @@ files:
 - lib/bootsnap/load_path_cache/path.rb
 - lib/bootsnap/load_path_cache/path_scanner.rb
 - lib/bootsnap/load_path_cache/store.rb
+- lib/bootsnap/rake.rb
 - lib/bootsnap/setup.rb
 - lib/bootsnap/version.rb
-homepage: https://github.com/Shopify/bootsnap
+homepage: https://github.com/rails/bootsnap
 licenses:
 - MIT
 metadata:
-  bug_tracker_uri: https://github.com/Shopify/bootsnap/issues
-  changelog_uri: https://github.com/Shopify/bootsnap/blob/main/CHANGELOG.md
-  source_code_uri: https://github.com/Shopify/bootsnap
+  bug_tracker_uri: https://github.com/rails/bootsnap/issues
+  changelog_uri: https://github.com/rails/bootsnap/blob/main/CHANGELOG.md
+  source_code_uri: https://github.com/rails/bootsnap
   allowed_push_host: https://rubygems.org
-post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -76,15 +73,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: 2.6.0
+      version: 2.7.0
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.5.16
-signing_key:
+rubygems_version: 3.6.9
 specification_version: 4
 summary: Boot large ruby/rails apps faster
 test_files: []

data/ext/bootsnap/bootsnap.h

@@ -1,6 +0,0 @@
-#ifndef BOOTSNAP_H
-#define BOOTSNAP_H 1
-
-/* doesn't expose anything */
-
-#endif /* BOOTSNAP_H */

data/lib/bootsnap/compile_cache/json.rb

@@ -1,89 +0,0 @@
-# frozen_string_literal: true
-
-require "bootsnap/bootsnap"
-
-module Bootsnap
-  module CompileCache
-    module JSON
-      class << self
-        attr_accessor(:msgpack_factory, :supported_options)
-        attr_reader(:cache_dir)
-
-        def cache_dir=(cache_dir)
-          @cache_dir = cache_dir.end_with?("/") ? "#{cache_dir}json" : "#{cache_dir}-json"
-        end
-
-        def input_to_storage(payload, _)
-          obj = ::JSON.parse(payload)
-          msgpack_factory.dump(obj)
-        end
-
-        def storage_to_output(data, kwargs)
-          if kwargs&.key?(:symbolize_names)
-            kwargs[:symbolize_keys] = kwargs.delete(:symbolize_names)
-          end
-          msgpack_factory.load(data, kwargs)
-        end
-
-        def input_to_output(data, kwargs)
-          ::JSON.parse(data, **(kwargs || {}))
-        end
-
-        def precompile(path)
-          Bootsnap::CompileCache::Native.precompile(
-            cache_dir,
-            path.to_s,
-            self,
-          )
-        end
-
-        def install!(cache_dir)
-          self.cache_dir = cache_dir
-          init!
-          if ::JSON.respond_to?(:load_file)
-            ::JSON.singleton_class.prepend(Patch)
-          end
-        end
-
-        def init!
-          require "json"
-          require "msgpack"
-
-          self.msgpack_factory = MessagePack::Factory.new
-          self.supported_options = [:symbolize_names]
-          if supports_freeze?
-            self.supported_options = [:freeze]
-          end
-          supported_options.freeze
-        end
-
-        private
-
-        def supports_freeze?
-          ::JSON.parse('["foo"]', freeze: true).first.frozen? &&
-            MessagePack.load(MessagePack.dump("foo"), freeze: true).frozen?
-        end
-      end
-
-      module Patch
-        def load_file(path, *args)
-          return super if args.size > 1
-
-          if (kwargs = args.first)
-            return super unless kwargs.is_a?(Hash)
-            return super unless (kwargs.keys - ::Bootsnap::CompileCache::JSON.supported_options).empty?
-          end
-
-          ::Bootsnap::CompileCache::Native.fetch(
-            Bootsnap::CompileCache::JSON.cache_dir,
-            File.realpath(path),
-            ::Bootsnap::CompileCache::JSON,
-            kwargs,
-          )
-        end
-
-        ruby2_keywords :load_file if respond_to?(:ruby2_keywords, true)
-      end
-    end
-  end
-end