bootsnap 1.18.4 → 1.19.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2b2271b2fa08edc313e43ec1359be530fcb6b206a7b4b63f37065290289cbc9a
-  data.tar.gz: c5838e7d01c33320e765a2bf668c68c09fc6b0938fb8789653b526cb3c22a244
+  metadata.gz: 71cd833fdf912a81069349b4de87a96cf15997710494eebb4535245aeeb02305
+  data.tar.gz: e502571bd2d4863d5f548ce50dcd6502cc18a51dcbf00a101ad1b65f47510534
 SHA512:
-  metadata.gz: 77168e700a54163bf4574fbf9f82fea681a01f7d0c5c3a55340d3dabc98959fce1176bebef4c4d25d18efa257ad5314f13ca978a77f5cb4288296c63a1b36871
-  data.tar.gz: cc770950cc3032a5033873457a23b19823c35d9d24285f3b13fb95848e51d8f34defca5396c7b545b5530381271979c93026401e4ac32d28ec0f89ceadd57823
+  metadata.gz: d5af0fb8cabbb09d026c8a73043c24d0669004a10edbf0ee3f5baaf8597ac99b7022c0bcaef1c25948dcd055cdd4dbac2ed614d9b2a5d6e58f1e94b6cc32f127
+  data.tar.gz: db4c64829ebc8177ed9a83bda8cf801e570f6d48db3e27ddd4522dbd48043a0d619615ba583be4a9ff1a24049785e5dfeed362f36ef1b9115662e9c3b612973d

data/CHANGELOG.md

@@ -1,5 +1,19 @@
 # Unreleased
 
+# 1.19.0
+
+* Remove JSON parsing cache. Recent versions of the `json` gem are as fast as `msgpack` if not faster.
+
+# 1.18.6
+
+* Fix cgroup CPU limits detection in CLI.
+
+# 1.18.5
+
+* Attempt to detect a QEMU bug that can cause `bootsnap precompile` to hang forever when building ARM64 docker images
+  from x86_64 machines. See #495.
+* Improve CLI to detect cgroup CPU limits and avoid spawning too many worker processes.
+
 # 1.18.4
 
 * Allow using bootsnap without bundler. See #488.
@@ -95,7 +109,7 @@
 
 * Get rid of the `Kernel.require_relative` decorator by resolving `$LOAD_PATH` members to their real path.
   This way we handle symlinks in `$LOAD_PATH` much more efficiently. See #402 for the detailed explanation.
-  
+
 * Drop support for Ruby 2.3 (to allow getting rid of the `Kernel.require_relative` decorator).
 
 # 1.10.3
@@ -221,7 +235,7 @@
 * Adds an instrumentation API to monitor cache misses.
 * Allow to control the behavior of `require 'bootsnap/setup'` using environment variables.
 * Deprecate the `disable_trace` option.
-* Deprecate the `ActiveSupport::Dependencies` (AKA Classic autoloader) integration. (#344) 
+* Deprecate the `ActiveSupport::Dependencies` (AKA Classic autoloader) integration. (#344)
 
 # 1.6.0
 
@@ -241,12 +255,12 @@
 
 # 1.4.9
 
-* [Windows support](https://github.com/Shopify/bootsnap/pull/319)
-* [Fix potential crash](https://github.com/Shopify/bootsnap/pull/322)
+* [Windows support](https://github.com/rails/bootsnap/pull/319)
+* [Fix potential crash](https://github.com/rails/bootsnap/pull/322)
 
 # 1.4.8
 
-* [Prevent FallbackScan from polluting exception cause](https://github.com/Shopify/bootsnap/pull/314)
+* [Prevent FallbackScan from polluting exception cause](https://github.com/rails/bootsnap/pull/314)
 
 # 1.4.7
 
@@ -259,7 +273,7 @@
   required if a different file with the same name was already being required
 
   Example:
-  
+
       require 'foo'
       require 'foo.en'
 
@@ -313,7 +327,7 @@
 
 # 1.3.0
 
-* Handle cases where load path entries are symlinked (https://github.com/Shopify/bootsnap/pull/136)
+* Handle cases where load path entries are symlinked (https://github.com/rails/bootsnap/pull/136)
 
 # 1.2.1
 

data/LICENSE.txt

@@ -1,6 +1,7 @@
 The MIT License (MIT)
 
-Copyright (c) 2017-present Shopify, Inc.
+Copyright (c) 2017-2025 Shopify, Inc.
+Copyright (c) 2025-present Rails Foundation
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

data/README.md

@@ -1,4 +1,4 @@
-# Bootsnap [![Actions Status](https://github.com/Shopify/bootsnap/workflows/ci/badge.svg)](https://github.com/Shopify/bootsnap/actions)
+# Bootsnap [![Actions Status](https://github.com/rails/bootsnap/workflows/ci/badge.svg)](https://github.com/rails/bootsnap/actions)
 
 Bootsnap is a library that plugs into Ruby, with optional support for `YAML` and `JSON`,
 to optimize and cache expensive computations. See [How Does This Work](#how-does-this-work).
@@ -41,7 +41,7 @@ getting progressively slower, this is almost certainly the cause.**
 It's technically possible to simply specify `gem 'bootsnap', require: 'bootsnap/setup'`, but it's
 important to load Bootsnap as early as possible to get maximum performance improvement.
 
-You can see how this require works [here](https://github.com/Shopify/bootsnap/blob/main/lib/bootsnap/setup.rb).
+You can see how this require works [here](https://github.com/rails/bootsnap/blob/main/lib/bootsnap/setup.rb).
 
 If you are not using Rails, or if you are but want more control over things, add this to your
 application setup immediately after `require 'bundler/setup'` (i.e. as early as possible: the sooner
@@ -57,13 +57,12 @@ Bootsnap.setup(
   load_path_cache:      true,                 # Optimize the LOAD_PATH with a cache
   compile_cache_iseq:   true,                 # Compile Ruby code into ISeq cache, breaks coverage reporting.
   compile_cache_yaml:   true,                 # Compile YAML into a cache
-  compile_cache_json:   true,                 # Compile JSON into a cache
   readonly:             true,                 # Use the caches but don't update them on miss or stale entries.
 )
 ```
 
 **Protip:** You can replace `require 'bootsnap'` with `BootLib::Require.from_gem('bootsnap',
-'bootsnap')` using [this trick](https://github.com/Shopify/bootsnap/wiki/Bootlib::Require). This
+'bootsnap')` using [this trick](https://github.com/rails/bootsnap/wiki/Bootlib::Require). This
 will help optimize boot time further if you have an extremely large `$LOAD_PATH`.
 
 Note: Bootsnap and [Spring](https://github.com/rails/spring) are orthogonal tools. While Bootsnap
@@ -170,7 +169,7 @@ The only directories considered "stable" are things under the Ruby install prefi
 "volatile".
 
 In addition to the [`Bootsnap::LoadPathCache::Cache`
-source](https://github.com/Shopify/bootsnap/blob/main/lib/bootsnap/load_path_cache/cache.rb),
+source](https://github.com/rails/bootsnap/blob/main/lib/bootsnap/load_path_cache/cache.rb),
 this diagram may help clarify how entry resolution works:
 
 ![How path searching works](https://cloud.githubusercontent.com/assets/3074765/25388270/670b5652-299b-11e7-87fb-975647f68981.png)
@@ -188,7 +187,7 @@ result too, raising a `LoadError` without touching the filesystem at all.
 
 Ruby has complex grammar and parsing it is not a particularly cheap operation. Since 1.9, Ruby has
 translated ruby source to an internal bytecode format, which is then executed by the Ruby VM. Since
-2.3.0, Ruby [exposes an API](https://ruby-doc.org/core-2.3.0/RubyVM/InstructionSequence.html) that
+2.3.0, Ruby [exposes an API](https://docs.ruby-lang.org/en/master/RubyVM/InstructionSequence.html) that
 allows caching that bytecode. This allows us to bypass the relatively-expensive compilation step on
 subsequent loads of the same file.
 
@@ -331,9 +330,23 @@ To do so you can use the `bootsnap precompile` command.
 Example:
 
 ```bash
-$ bundle exec bootsnap precompile --gemfile app/ lib/
+$ bundle exec bootsnap precompile --gemfile app/ lib/ config/
 ```
 
+## Known issues
+
+### QEMU environments
+
+When building cross-platform Docker images, QEMU is often used for emulation and can be the source of a limitation that causes forked processes to hang. While Bootsnap includes automatic detection for this issue (as of [PR #501](https://github.com/rails/bootsnap/pull/501)), the detection may not always be sufficient.
+
+If you encounter hangs during precompilation in QEMU-based environments (such as when using Docker buildx for cross-platform builds), you can work around this by disabling parallelization with the `-j 0` option:
+
+```bash
+$ bundle exec bootsnap precompile -j 0 --gemfile app/ lib/ config/
+```
+
+See [Issue #495](https://github.com/rails/bootsnap/issues/495) for more details about this QEMU-related issue.
+
 ## When not to use Bootsnap
 
 *Alternative engines*: Bootsnap is pretty reliant on MRI features, and parts are disabled entirely on alternative ruby

data/ext/bootsnap/bootsnap.c

@@ -23,7 +23,7 @@
 
 #ifdef __APPLE__
   // The symbol is present, however not in the headers
-  // See: https://github.com/Shopify/bootsnap/issues/470
+  // See: https://github.com/rails/bootsnap/issues/470
   extern int fdatasync(int);
 #endif
 

data/ext/bootsnap/extconf.rb

@@ -12,7 +12,7 @@ if %w[ruby truffleruby].include?(RUBY_ENGINE)
   append_cflags ["-O3", "-std=c99"]
 
   # ruby.h has some -Wpedantic fails in some cases
-  # (e.g. https://github.com/Shopify/bootsnap/issues/15)
+  # (e.g. https://github.com/rails/bootsnap/issues/15)
   unless ["0", "", nil].include?(ENV["BOOTSNAP_PEDANTIC"])
     append_cflags([
       "-Wall",

data/lib/bootsnap/cli/worker_pool.rb

@@ -1,16 +1,88 @@
 # frozen_string_literal: true
 
+require "etc"
+require "rbconfig"
+require "io/wait" unless IO.method_defined?(:wait_readable)
+
 module Bootsnap
   class CLI
     class WorkerPool
       class << self
         def create(size:, jobs:)
+          size ||= default_size
           if size > 0 && Process.respond_to?(:fork)
             new(size: size, jobs: jobs)
           else
             Inline.new(jobs: jobs)
           end
         end
+
+        def default_size
+          nprocessors = Etc.nprocessors
+          size = [nprocessors, cpu_quota&.to_i || nprocessors].min
+          case size
+          when 0, 1
+            0
+          else
+            if fork_defunct?
+              $stderr.puts "warning: faulty fork(2) detected, probably in cross platform docker builds. " \
+                           "Disabling parallel compilation."
+              0
+            else
+              size
+            end
+          end
+        end
+
+        def cpu_quota
+          if RbConfig::CONFIG["target_os"].include?("linux")
+            if File.exist?("/sys/fs/cgroup/cpu.max")
+              # cgroups v2: https://docs.kernel.org/admin-guide/cgroup-v2.html#cpu-interface-files
+              cpu_max = File.read("/sys/fs/cgroup/cpu.max")
+              return nil if cpu_max.start_with?("max ") # no limit
+
+              max, period = cpu_max.split.map(&:to_f)
+              max / period
+            elsif File.exist?("/sys/fs/cgroup/cpu,cpuacct/cpu.cfs_quota_us")
+              # cgroups v1: https://kernel.googlesource.com/pub/scm/linux/kernel/git/glommer/memcg/+/cpu_stat/Documentation/cgroups/cpu.txt
+              max = File.read("/sys/fs/cgroup/cpu,cpuacct/cpu.cfs_quota_us").to_i
+              # If the cpu.cfs_quota_us is -1, cgroup does not adhere to any CPU time restrictions
+              # https://docs.kernel.org/scheduler/sched-bwc.html#management
+              return nil if max <= 0
+
+              period = File.read("/sys/fs/cgroup/cpu,cpuacct/cpu.cfs_period_us").to_f
+              max / period
+            end
+          end
+        end
+
+        def fork_defunct?
+          return true unless ::Process.respond_to?(:fork)
+
+          # Ref: https://github.com/rails/bootsnap/issues/495
+          # The second forked process will hang on some QEMU environments
+          r, w = IO.pipe
+          pids = 2.times.map do
+            ::Process.fork do
+              exit!(true)
+            end
+          end
+          w.close
+          r.wait_readable(1) # Wait at most 1s
+
+          defunct = false
+
+          pids.each do |pid|
+            _pid, status = ::Process.wait2(pid, ::Process::WNOHANG)
+            if status.nil? # Didn't exit in 1s
+              defunct = true
+              Process.kill(:KILL, pid)
+              ::Process.wait2(pid)
+            end
+          end
+
+          defunct
+        end
       end
 
       class Inline

data/lib/bootsnap/cli.rb

@@ -4,7 +4,6 @@ require "bootsnap"
 require "bootsnap/cli/worker_pool"
 require "optparse"
 require "fileutils"
-require "etc"
 
 module Bootsnap
   class CLI
@@ -21,7 +20,7 @@ module Bootsnap
 
     attr_reader :cache_dir, :argv
 
-    attr_accessor :compile_gemfile, :exclude, :verbose, :iseq, :yaml, :json, :jobs
+    attr_accessor :compile_gemfile, :exclude, :verbose, :iseq, :yaml, :jobs
 
     def initialize(argv)
       @argv = argv
@@ -29,10 +28,9 @@ module Bootsnap
       self.compile_gemfile = false
       self.exclude = nil
       self.verbose = false
-      self.jobs = Etc.nprocessors
+      self.jobs = nil
       self.iseq = true
       self.yaml = true
-      self.json = true
     end
 
     def precompile_command(*sources)
@@ -43,21 +41,18 @@ module Bootsnap
           cache_dir: cache_dir,
           iseq: iseq,
           yaml: yaml,
-          json: json,
           revalidation: true,
         )
 
         @work_pool = WorkerPool.create(size: jobs, jobs: {
           ruby: method(:precompile_ruby),
           yaml: method(:precompile_yaml),
-          json: method(:precompile_json),
         })
         @work_pool.spawn
 
         main_sources = sources.map { |d| File.expand_path(d) }
         precompile_ruby_files(main_sources)
         precompile_yaml_files(main_sources)
-        precompile_json_files(main_sources)
 
         if compile_gemfile
           # Gems that include JSON or YAML files usually don't put them in `lib/`.
@@ -71,7 +66,6 @@ module Bootsnap
 
           precompile_ruby_files(gem_paths, exclude: gem_exclude)
           precompile_yaml_files(gem_paths, exclude: gem_exclude)
-          precompile_json_files(gem_paths, exclude: gem_exclude)
         end
 
         if (exitstatus = @work_pool.shutdown)
@@ -146,29 +140,6 @@ module Bootsnap
       end
     end
 
-    def precompile_json_files(load_paths, exclude: self.exclude)
-      return unless json
-
-      load_paths.each do |path|
-        if !exclude || !exclude.match?(path)
-          list_files(path, "**/*.json").each do |json_file|
-            # We ignore hidden files to not match the various .config.json files
-            if !File.basename(json_file).start_with?(".") && (!exclude || !exclude.match?(json_file))
-              @work_pool.push(:json, json_file)
-            end
-          end
-        end
-      end
-    end
-
-    def precompile_json(*json_files)
-      Array(json_files).each do |json_file|
-        if CompileCache::JSON.precompile(json_file) && verbose
-          $stderr.puts(json_file)
-        end
-      end
-    end
-
     def precompile_ruby_files(load_paths, exclude: self.exclude)
       return unless iseq
 
@@ -277,9 +248,9 @@ module Bootsnap
         opts.on("--no-yaml", help) { self.yaml = false }
 
         help = <<~HELP
-          Disable JSON precompilation.
+          Disable JSON precompilation. Deprecated.
         HELP
-        opts.on("--no-json", help) { self.json = false }
+        opts.on("--no-json", help) { $stderr.puts("The --no-json option is deprecated and now a noop.") }
       end
     end
   end

data/lib/bootsnap/compile_cache.rb

@@ -9,7 +9,11 @@ module Bootsnap
 
     Error = Class.new(StandardError)
 
-    def self.setup(cache_dir:, iseq:, yaml:, json:, readonly: false, revalidation: false)
+    def self.setup(cache_dir:, iseq:, yaml:, json: (json_unset = true), readonly: false, revalidation: false)
+      unless json_unset
+        warn("Bootsnap::CompileCache.setup `json` argument is deprecated and has no effect")
+      end
+
       if iseq
         if supported?
           require_relative "compile_cache/iseq"
@@ -28,15 +32,6 @@ module Bootsnap
         end
       end
 
-      if json
-        if supported?
-          require_relative "compile_cache/json"
-          Bootsnap::CompileCache::JSON.install!(cache_dir)
-        elsif $VERBOSE
-          warn("[bootsnap/setup] JSON parsing caching is not supported on this implementation of Ruby")
-        end
-      end
-
       if supported? && defined?(Bootsnap::CompileCache::Native)
         Bootsnap::CompileCache::Native.readonly = readonly
         Bootsnap::CompileCache::Native.revalidation = revalidation

data/lib/bootsnap/load_path_cache/loaded_features_index.rb

@@ -142,7 +142,7 @@ module Bootsnap
       # will _never_ run on MacOS, and therefore think they can get away
       # with calling a Ruby file 'x.dylib.rb' and then requiring it as 'x.dylib'.)
       #
-      # See <https://ruby-doc.org/core-2.6.4/Kernel.html#method-i-require>.
+      # See <https://docs.ruby-lang.org/en/master/Kernel.html#method-i-require>.
       def extension_elidable?(feature)
         feature.to_s.end_with?(".rb", ".so", ".o", ".dll", ".dylib")
       end

data/lib/bootsnap/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Bootsnap
-  VERSION = "1.18.4"
+  VERSION = "1.19.0"
 end

data/lib/bootsnap.rb

@@ -54,8 +54,12 @@ module Bootsnap
       revalidation: false,
       compile_cache_iseq: true,
       compile_cache_yaml: true,
-      compile_cache_json: true
+      compile_cache_json: (compile_cache_json_unset = true)
     )
+      unless compile_cache_json_unset
+        warn("Bootsnap.setup `compile_cache_json` argument is deprecated and has no effect")
+      end
+
       if load_path_cache
         Bootsnap::LoadPathCache.setup(
           cache_path: "#{cache_dir}/bootsnap/load-path-cache",
@@ -69,7 +73,6 @@ module Bootsnap
         cache_dir: "#{cache_dir}/bootsnap/compile-cache",
         iseq: compile_cache_iseq,
         yaml: compile_cache_yaml,
-        json: compile_cache_json,
         readonly: readonly,
         revalidation: revalidation,
       )
@@ -115,7 +118,6 @@ module Bootsnap
           load_path_cache: enabled?("BOOTSNAP_LOAD_PATH_CACHE"),
           compile_cache_iseq: enabled?("BOOTSNAP_COMPILE_CACHE"),
           compile_cache_yaml: enabled?("BOOTSNAP_COMPILE_CACHE"),
-          compile_cache_json: enabled?("BOOTSNAP_COMPILE_CACHE"),
           readonly: bool_env("BOOTSNAP_READONLY"),
           revalidation: bool_env("BOOTSNAP_REVALIDATE"),
           ignore_directories: ignore_directories,

metadata

@@ -1,14 +1,13 @@
 --- !ruby/object:Gem::Specification
 name: bootsnap
 version: !ruby/object:Gem::Version
-  version: 1.18.4
+  version: 1.19.0
 platform: ruby
 authors:
 - Burke Libbey
-autorequire:
 bindir: exe
 cert_chain: []
-date: 2024-08-05 00:00:00.000000000 Z
+date: 1980-01-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: msgpack
@@ -46,7 +45,6 @@ files:
 - lib/bootsnap/cli/worker_pool.rb
 - lib/bootsnap/compile_cache.rb
 - lib/bootsnap/compile_cache/iseq.rb
-- lib/bootsnap/compile_cache/json.rb
 - lib/bootsnap/compile_cache/yaml.rb
 - lib/bootsnap/explicit_require.rb
 - lib/bootsnap/load_path_cache.rb
@@ -60,15 +58,14 @@ files:
 - lib/bootsnap/load_path_cache/store.rb
 - lib/bootsnap/setup.rb
 - lib/bootsnap/version.rb
-homepage: https://github.com/Shopify/bootsnap
+homepage: https://github.com/rails/bootsnap
 licenses:
 - MIT
 metadata:
-  bug_tracker_uri: https://github.com/Shopify/bootsnap/issues
-  changelog_uri: https://github.com/Shopify/bootsnap/blob/main/CHANGELOG.md
-  source_code_uri: https://github.com/Shopify/bootsnap
+  bug_tracker_uri: https://github.com/rails/bootsnap/issues
+  changelog_uri: https://github.com/rails/bootsnap/blob/main/CHANGELOG.md
+  source_code_uri: https://github.com/rails/bootsnap
   allowed_push_host: https://rubygems.org
-post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -83,8 +80,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.5.16
-signing_key:
+rubygems_version: 3.6.9
 specification_version: 4
 summary: Boot large ruby/rails apps faster
 test_files: []

data/lib/bootsnap/compile_cache/json.rb

@@ -1,89 +0,0 @@
-# frozen_string_literal: true
-
-require "bootsnap/bootsnap"
-
-module Bootsnap
-  module CompileCache
-    module JSON
-      class << self
-        attr_accessor(:msgpack_factory, :supported_options)
-        attr_reader(:cache_dir)
-
-        def cache_dir=(cache_dir)
-          @cache_dir = cache_dir.end_with?("/") ? "#{cache_dir}json" : "#{cache_dir}-json"
-        end
-
-        def input_to_storage(payload, _)
-          obj = ::JSON.parse(payload)
-          msgpack_factory.dump(obj)
-        end
-
-        def storage_to_output(data, kwargs)
-          if kwargs&.key?(:symbolize_names)
-            kwargs[:symbolize_keys] = kwargs.delete(:symbolize_names)
-          end
-          msgpack_factory.load(data, kwargs)
-        end
-
-        def input_to_output(data, kwargs)
-          ::JSON.parse(data, **(kwargs || {}))
-        end
-
-        def precompile(path)
-          Bootsnap::CompileCache::Native.precompile(
-            cache_dir,
-            path.to_s,
-            self,
-          )
-        end
-
-        def install!(cache_dir)
-          self.cache_dir = cache_dir
-          init!
-          if ::JSON.respond_to?(:load_file)
-            ::JSON.singleton_class.prepend(Patch)
-          end
-        end
-
-        def init!
-          require "json"
-          require "msgpack"
-
-          self.msgpack_factory = MessagePack::Factory.new
-          self.supported_options = [:symbolize_names]
-          if supports_freeze?
-            self.supported_options = [:freeze]
-          end
-          supported_options.freeze
-        end
-
-        private
-
-        def supports_freeze?
-          ::JSON.parse('["foo"]', freeze: true).first.frozen? &&
-            MessagePack.load(MessagePack.dump("foo"), freeze: true).frozen?
-        end
-      end
-
-      module Patch
-        def load_file(path, *args)
-          return super if args.size > 1
-
-          if (kwargs = args.first)
-            return super unless kwargs.is_a?(Hash)
-            return super unless (kwargs.keys - ::Bootsnap::CompileCache::JSON.supported_options).empty?
-          end
-
-          ::Bootsnap::CompileCache::Native.fetch(
-            Bootsnap::CompileCache::JSON.cache_dir,
-            File.realpath(path),
-            ::Bootsnap::CompileCache::JSON,
-            kwargs,
-          )
-        end
-
-        ruby2_keywords :load_file if respond_to?(:ruby2_keywords, true)
-      end
-    end
-  end
-end