bootsnap 1.4.5 → 1.18.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: be54a60d54f32f824d5c2fdccf189cdcb5409d1848c55f6e654fe44ace1d3f21
-  data.tar.gz: c6239c30984a936b76e218c2b1292a7e72d817d3e4847a6c58f0121c1e3068dc
+  metadata.gz: 4fa4ab785277ee01a1c8ee75b43f0efb93db42bffcdacc1c8505a65efa03dede
+  data.tar.gz: 8aaaca48ae257b563580023c8fa36a59463f4c30f5463c14f6b8b94bf5fe27df
 SHA512:
-  metadata.gz: c251990457406cd51726eec8d62e0e1028bbeade6e24f266cb743d6f00f53650841beed9ee8e1795c78febd18ea234691125920c16aa5a95031718a28619bb31
-  data.tar.gz: 554a885bf2264f088618c41864fb84069d0664b35af0ae4b619c8fbb1cf285c80fad564d36845161f8044c35c6b53a33e66f27eb75c6715f25869af2795c97f6
+  metadata.gz: 27b48d27d3330c8565952a2fbb979e71013b1e9585bcb3284656192808c304f2874c32a135b14895eec61a7ef038fa71fa111964a56e7aaedc9ff507ef307686
+  data.tar.gz: c3d83a0b068f2908a6298c7cd8e1a660f1228a7ddbfb9409cb3f6c174319f3974388ce73756c04062b17b97a37e199a07bccbb0b8dc1c6224998c58c51194b27

data/CHANGELOG.md

@@ -1,3 +1,267 @@
+# Unreleased
+
+# 1.18.3
+
+* Fix the cache corruption issue in the revalidation feature. See #474.
+  The cache revalidation feature remains opt-in for now, until it is more battle tested.
+
+# 1.18.2
+
+* Disable stale cache entries revalidation by default as it seems to cause cache corruption issues. See #471 and #474.
+  Will be re-enabled in a future version once the root cause is identified.
+* Fix a potential compilation issue on some systems. See #470.
+
+# 1.18.1
+
+* Handle `EPERM` errors when opening files with `O_NOATIME`.
+
+# 1.18.0
+
+* `Bootsnap.instrumentation` now receive `:hit` events.
+* Add `Bootsnap.log_stats!` to print hit rate statistics on process exit. Can also be enabled with `BOOTSNAP_STATS=1`.
+* Revalidate stale cache entries by digesting the source content.
+  This should significantly improve performance in environments where `mtime` isn't preserved (e.g. CI systems doing a git clone, etc).
+  See #468.
+* Open source files and cache entries with `O_NOATIME` when available to reduce disk accesses. See #469.
+* `bootsnap precompile --gemfile` now look for `.rb` files in the whole gem and not just the `lib/` directory. See #466.
+
+# 1.17.1
+
+* Fix a compatibility issue with the `prism` library that ships with Ruby 3.3. See #463.
+* Improved the `Kernel#require` decorator to not cause a method redefinition warning. See #461.
+
+# 1.17.0
+
+* Ensure `$LOAD_PATH.dup` is Ractor shareable to fix an conflict with `did_you_mean`.
+* Allow to ignore directories using absolute paths.
+* Support YAML and JSON CompileCache on TruffleRuby.
+* Support LoadPathCache on TruffleRuby.
+
+# 1.16.0
+
+* Use `RbConfig::CONFIG["rubylibdir"]` instead of `RbConfig::CONFIG["libdir"]` to check for stdlib files. See #431.
+* Fix the cached version of `YAML.load_file` being slightly more permissive than the default `Psych` one. See #434.
+  `Date` and `Time` values are now properly rejected, as well as aliases.
+  If this causes a regression in your application, it is recommended to load *trusted* YAML files with `YAML.unsafe_load_file`.
+
+# 1.15.0
+
+* Add a readonly mode, for environments in which the updated cache wouldn't be persisted. See #428 and #423.
+
+# 1.14.0
+
+* Require Ruby 2.6.
+* Add a way to skip directories during load path scanning.
+  If you have large non-ruby directories in the middle of your load path, it can severely slow down scanning.
+  Typically this is a problem with `node_modules`. See #277.
+* Fix `Bootsnap.unload_cache!`, it simply wouldn't work at all because of a merge mistake. See #421.
+
+# 1.13.0
+
+* Stop decorating `Kernel.load`. This used to be very useful in development because the Rails "classic" autoloader
+  was using `Kernel.load` in dev and `Kernel.require` in production. But Zeitwerk is now the default, and it doesn't
+  use `Kernel.load` at all.
+
+  People still using the classic autoloader might want to stick to `bootsnap 1.12`.
+
+* Add `Bootsnap.unload_cache!`. Applications can call it at the end of their boot sequence when they know
+  no more code will be loaded to reclaim a bit of memory.
+
+# 1.12.0
+
+* `bootsnap precompile` CLI will now also precompile `Rakefile` and `.rake` files.
+
+* Stop decorating `Module#autoload` as it was only useful for supporting Ruby 2.2 and older.
+
+* Remove `uname` and other platform specific version from the cache keys. `RUBY_PLATFORM + RUBY_REVISION` should be
+  enough to ensure bytecode compatibility. This should improve caching for alpine based setups. See #409.
+
+# 1.11.1
+
+* Fix the `can't modify frozen Hash` error on load path cache mutation. See #411.
+
+# 1.11.0
+
+* Drop dependency on `fileutils`.
+
+* Better respect `Kernel#require` duck typing. While it almost never comes up in practice, `Kernel#require`
+  follow a fairly intricate duck-typing protocol on its argument implemented as `rb_get_path(VALUE)` in MRI.
+  So when applicable we bind `rb_get_path` and use it for improved compatibility. See #396 and #406.
+
+* Get rid of the `Kernel.require_relative` decorator by resolving `$LOAD_PATH` members to their real path.
+  This way we handle symlinks in `$LOAD_PATH` much more efficiently. See #402 for the detailed explanation.
+  
+* Drop support for Ruby 2.3 (to allow getting rid of the `Kernel.require_relative` decorator).
+
+# 1.10.3
+
+* Fix Regexp and Date type support in YAML compile cache. (#400)
+
+* Improve the YAML compile cache to support `UTF-8` symbols. (#398, #399)
+  [The default `MessagePack` symbol serializer assumes all symbols are ASCII](https://github.com/msgpack/msgpack-ruby/pull/211),
+  because of this, non-ASCII compatible symbol would be restored with `ASCII_8BIT` encoding (AKA `BINARY`).
+  Bootsnap now properly cache them in `UTF-8`.
+
+  Note that the above only apply for actual YAML symbols (e..g `--- :foo`).
+  The issue is still present for string keys parsed with `YAML.load_file(..., symbolize_names: true)`, that is a bug
+  in `msgpack` that will hopefully be solved soon, see: https://github.com/msgpack/msgpack-ruby/pull/246
+
+* Entirely disable the YAML compile cache if `Encoding.default_internal` is set to an encoding not supported by `msgpack`. (#398)
+  `Psych` coerce strings to `Encoding.default_internal`, but `MessagePack` doesn't. So in this scenario we can't provide
+  YAML caching at all without returning the strings in the wrong encoding.
+  This never came up in practice but might as well be safe.
+
+# 1.10.2
+
+* Reduce the `Kernel.require` extra stack frames some more. Now bootsnap should only add one extra frame per `require` call.
+
+* Better check `freeze` option support in JSON compile cache.
+  Previously `JSON.load_file(..., freeze: true)` would be cached even when the msgpack version is missing support for it.
+
+# 1.10.1
+
+* Fix `Kernel#autoload`'s fallback path always being executed.
+
+* Consider `unlink` failing with `ENOENT` as a success.
+
+# 1.10.0
+
+* Delay requiring `FileUtils`. (#285)
+  `FileUtils` can be installed as a gem, so it's best to wait for bundler to have setup the load path before requiring it.
+
+* Improve support of Psych 4. (#392)
+  Since `1.8.0`, `YAML.load_file` was no longer cached when Psych 4 was used. This is because `load_file` loads
+  in safe mode by default, so the Bootsnap cache could defeat that safety.
+  Now when precompiling YAML files, Bootsnap first try to parse them in safe mode, and if it can't fallback to unsafe mode,
+  and the cache contains a flag that records whether it was generated in safe mode or not.
+  `YAML.unsafe_load_file` will use safe caches just fine, but `YAML.load_file` will fallback to uncached YAML parsing
+  if the cache was generated using unsafe parsing.
+
+* Minimize the Kernel.require extra stack frames. (#393)
+  This should reduce the noise generated by bootsnap on `LoadError`.
+
+# 1.9.4
+
+* Ignore absolute paths in the loaded feature index. (#385)
+  This fixes a compatibility issue with Zeitwerk when Zeitwerk is loaded before bootsnap. It also should
+  reduce the memory usage and improve load performance of Zeitwerk managed files.
+
+* Automatically invalidate the load path cache whenever the Ruby version change. (#387)
+  This is to avoid issues in case the same installation path is re-used for subsequent ruby patch releases.
+
+# 1.9.3
+
+* Only disable the compile cache for source files impacted by [Ruby 3.0.3 [Bug 18250]](https://bugs.ruby-lang.org/issues/18250).
+  This should keep the performance loss to a minimum.
+
+# 1.9.2
+
+* Disable compile cache if [Ruby 3.0.3's ISeq cache bug](https://bugs.ruby-lang.org/issues/18250) is detected.
+  AKA `iseq.rb:13 to_binary: wrong argument type false (expected Symbol)`
+* Fix `Kernel.load` behavior: before `load 'a'` would load `a.rb` (and other tried extensions) and wouldn't load `a` unless `development_mode: true`, now only `a` would be loaded and files with extensions wouldn't be.
+
+# 1.9.1
+
+* Removed a forgotten debug statement in JSON precompilation.
+
+# 1.9.0
+
+* Added a compilation cache for `JSON.load_file`. (#370)
+
+# 1.8.1
+
+* Fixed support for older Psych. (#369)
+
+# 1.8.0
+
+* Improve support for Psych 4. (#368)
+
+# 1.7.7
+
+* Fix `require_relative` in evaled code on latest ruby 3.1.0-dev. (#366)
+
+# 1.7.6
+
+* Fix reliance on `set` to be required.
+* Fix `Encoding::UndefinedConversionError` error for Rails applications when precompiling cache. (#364)
+
+# 1.7.5
+
+* Handle a regression of Ruby 2.7.3 causing Bootsnap to call the deprecated `untaint` method. (#360)
+* Gracefully handle read-only file system as well as other errors preventing to persist the load path cache. (#358)
+
+# 1.7.4
+
+* Stop raising errors when encountering various file system errors. The cache is now best effort,
+  if somehow it can't be saved, bootsnap will gracefully fallback to the original operation (e.g. `Kernel.require`).
+  (#353, #177, #262)
+
+# 1.7.3
+
+* Disable YAML precompilation when encountering YAML tags. (#351)
+
+# 1.7.2
+
+* Fix compatibility with msgpack < 1. (#349)
+
+# 1.7.1
+
+* Warn Ruby 2.5 users if they turn ISeq caching on. (#327, #244)
+* Disable ISeq caching for the whole 2.5.x series again.
+* Better handle hashing of Ruby strings. (#318)
+
+# 1.7.0
+
+* Fix detection of YAML files in gems.
+* Adds an instrumentation API to monitor cache misses.
+* Allow to control the behavior of `require 'bootsnap/setup'` using environment variables.
+* Deprecate the `disable_trace` option.
+* Deprecate the `ActiveSupport::Dependencies` (AKA Classic autoloader) integration. (#344) 
+
+# 1.6.0
+
+* Fix a Ruby 2.7/3.0 issue with `YAML.load_file` keyword arguments. (#342)
+* `bootsnap precompile` CLI use multiple processes to complete faster. (#341)
+* `bootsnap precompile` CLI also precompile YAML files. (#340)
+* Changed the load path cache directory from `$BOOTSNAP_CACHE_DIR/bootsnap-load-path-cache` to `$BOOTSNAP_CACHE_DIR/bootsnap/load-path-cache` for ease of use. (#334)
+* Changed the compile cache directory from `$BOOTSNAP_CACHE_DIR/bootsnap-compile-cache` to `$BOOTSNAP_CACHE_DIR/bootsnap/compile-cache` for ease of use. (#334)
+
+# 1.5.1
+
+* Workaround a Ruby bug in InstructionSequence.compile_file. (#332)
+
+# 1.5.0
+
+* Add a command line to statically precompile the ISeq cache. (#326)
+
+# 1.4.9
+
+* [Windows support](https://github.com/Shopify/bootsnap/pull/319)
+* [Fix potential crash](https://github.com/Shopify/bootsnap/pull/322)
+
+# 1.4.8
+
+* [Prevent FallbackScan from polluting exception cause](https://github.com/Shopify/bootsnap/pull/314)
+
+# 1.4.7
+
+* Various performance enhancements
+* Fix race condition in heavy concurrent load scenarios that would cause bootsnap to raise
+
+# 1.4.6
+
+* Fix bug that was erroneously considering that files containing `.` in the names were being
+  required if a different file with the same name was already being required
+
+  Example:
+  
+      require 'foo'
+      require 'foo.en'
+
+  Before bootsnap was considering `foo.en` to be the same file as `foo`
+
+* Use glibc as part of the ruby_platform cache key
+
 # 1.4.5
 
 * MRI 2.7 support

data/LICENSE.txt

@@ -1,6 +1,6 @@
 The MIT License (MIT)
 
-Copyright (c) 2017 Shopify, Inc.
+Copyright (c) 2017-present Shopify, Inc.
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

data/README.md

@@ -1,6 +1,6 @@
-# Bootsnap [![Build Status](https://travis-ci.org/Shopify/bootsnap.svg?branch=master)](https://travis-ci.org/Shopify/bootsnap)
+# Bootsnap [![Actions Status](https://github.com/Shopify/bootsnap/workflows/ci/badge.svg)](https://github.com/Shopify/bootsnap/actions)
 
-Bootsnap is a library that plugs into Ruby, with optional support for `ActiveSupport` and `YAML`,
+Bootsnap is a library that plugs into Ruby, with optional support for `YAML` and `JSON`,
 to optimize and cache expensive computations. See [How Does This Work](#how-does-this-work).
 
 #### Performance
@@ -11,7 +11,7 @@ to optimize and cache expensive computations. See [How Does This Work](#how-does
 - The core Shopify platform -- a rather large monolithic application -- boots about 75% faster,
   dropping from around 25s to 6.5s.
 * In Shopify core (a large app), about 25% of this gain can be attributed to `compile_cache_*`
-  features; 75% to path caching, and ~1% to `disable_trace`. This is fairly representative.
+  features; 75% to path caching. This is fairly representative.
 
 ## Usage
 
@@ -29,7 +29,8 @@ If you are using Rails, add this to `config/boot.rb` immediately after `require
 require 'bootsnap/setup'
 ```
 
-Note that bootsnap writes to `tmp/cache`, and that directory *must* be writable. Rails will fail to
+Note that bootsnap writes to `tmp/cache` (or the path specified by `ENV['BOOTSNAP_CACHE_DIR']`),
+and that directory *must* be writable. Rails will fail to
 boot if it is not. If this is unacceptable (e.g. you are running in a read-only container and
 unwilling to mount in a writable tmpdir), you should remove this line or wrap it in a conditional.
 
@@ -40,7 +41,7 @@ getting progressively slower, this is almost certainly the cause.**
 It's technically possible to simply specify `gem 'bootsnap', require: 'bootsnap/setup'`, but it's
 important to load Bootsnap as early as possible to get maximum performance improvement.
 
-You can see how this require works [here](https://github.com/Shopify/bootsnap/blob/master/lib/bootsnap/setup.rb).
+You can see how this require works [here](https://github.com/Shopify/bootsnap/blob/main/lib/bootsnap/setup.rb).
 
 If you are not using Rails, or if you are but want more control over things, add this to your
 application setup immediately after `require 'bundler/setup'` (i.e. as early as possible: the sooner
@@ -51,17 +52,16 @@ require 'bootsnap'
 env = ENV['RAILS_ENV'] || "development"
 Bootsnap.setup(
   cache_dir:            'tmp/cache',          # Path to your cache
+  ignore_directories:   ['node_modules'],     # Directory names to skip.
   development_mode:     env == 'development', # Current working environment, e.g. RACK_ENV, RAILS_ENV, etc
   load_path_cache:      true,                 # Optimize the LOAD_PATH with a cache
-  autoload_paths_cache: true,                 # Optimize ActiveSupport autoloads with cache
-  disable_trace:        true,                 # Set `RubyVM::InstructionSequence.compile_option = { trace_instruction: false }`
   compile_cache_iseq:   true,                 # Compile Ruby code into ISeq cache, breaks coverage reporting.
-  compile_cache_yaml:   true                  # Compile YAML into a cache
+  compile_cache_yaml:   true,                 # Compile YAML into a cache
+  compile_cache_json:   true,                 # Compile JSON into a cache
+  readonly:             true,                 # Use the caches but don't update them on miss or stale entries.
 )
 ```
 
-**Note that `disable_trace` will break debuggers and tracing.**
-
 **Protip:** You can replace `require 'bootsnap'` with `BootLib::Require.from_gem('bootsnap',
 'bootsnap')` using [this trick](https://github.com/Shopify/bootsnap/wiki/Bootlib::Require). This
 will help optimize boot time further if you have an extremely large `$LOAD_PATH`.
@@ -69,7 +69,22 @@ will help optimize boot time further if you have an extremely large `$LOAD_PATH`
 Note: Bootsnap and [Spring](https://github.com/rails/spring) are orthogonal tools. While Bootsnap
 speeds up the loading of individual source files, Spring keeps a copy of a pre-booted Rails process
 on hand to completely skip parts of the boot process the next time it's needed. The two tools work
-well together, and are both included in a newly-generated Rails applications by default.
+well together.
+
+### Environment variables
+
+`require 'bootsnap/setup'` behavior can be changed using environment variables:
+
+- `BOOTSNAP_CACHE_DIR` allows to define the cache location.
+- `DISABLE_BOOTSNAP` allows to entirely disable bootsnap.
+- `DISABLE_BOOTSNAP_LOAD_PATH_CACHE` allows to disable load path caching.
+- `DISABLE_BOOTSNAP_COMPILE_CACHE` allows to disable ISeq and YAML caches.
+- `BOOTSNAP_READONLY` configure bootsnap to not update the cache on miss or stale entries.
+- `BOOTSNAP_LOG` configure bootsnap to log all caches misses to STDERR.
+- `BOOTSNAP_STATS` log hit rate statistics on exit. Can't be used if `BOOTSNAP_LOG` is enabled.
+- `BOOTSNAP_IGNORE_DIRECTORIES` a comma separated list of directories that shouldn't be scanned.
+  Useful when you have large directories of non-ruby files inside `$LOAD_PATH`.
+  It defaults to ignore any directory named `node_modules`.
 
 ### Environments
 
@@ -77,6 +92,23 @@ All Bootsnap features are enabled in development, test, production, and all othe
 
 If you would like to disable any feature for a certain environment, we suggest changing the configuration to take into account the appropriate ENV var or configuration according to your needs.
 
+### Instrumentation
+
+Bootsnap cache misses can be monitored though a callback:
+
+```ruby
+Bootsnap.instrumentation = ->(event, path) { puts "#{event} #{path}" }
+```
+
+`event` is either `:hit`, `:miss`, `:stale` or `:revalidated`.
+You can also call `Bootsnap.log!` as a shortcut to log all events to STDERR.
+
+To turn instrumentation back off you can set it to nil:
+
+```ruby
+Bootsnap.instrumentation = nil
+```
+
 ## How does this work?
 
 Bootsnap optimizes methods to cache results of expensive computations, and can be grouped
@@ -84,13 +116,12 @@ into two broad categories:
 
 * [Path Pre-Scanning](#path-pre-scanning)
     * `Kernel#require` and `Kernel#load` are modified to eliminate `$LOAD_PATH` scans.
-    * `ActiveSupport::Dependencies.{autoloadable_module?,load_missing_constant,depend_on}` are
-      overridden to eliminate scans of `ActiveSupport::Dependencies.autoload_paths`.
 * [Compilation caching](#compilation-caching)
     * `RubyVM::InstructionSequence.load_iseq` is implemented to cache the result of ruby bytecode
       compilation.
     * `YAML.load_file` is modified to cache the result of loading a YAML object in MessagePack format
       (or Marshal, if the message uses types unsupported by MessagePack).
+    * `JSON.load_file` is modified to cache the result of loading a JSON object in MessagePack format
 
 ### Path Pre-Scanning
 
@@ -124,10 +155,6 @@ open y/foo.rb
 ...
 ```
 
-Exactly the same strategy is employed for methods that traverse
-`ActiveSupport::Dependencies.autoload_paths` if the `autoload_paths_cache` option is given to
-`Bootsnap.setup`.
-
 The following diagram flowcharts the overrides that make the `*_path_cache` features work.
 
 ![Flowchart explaining
@@ -143,7 +170,7 @@ The only directories considered "stable" are things under the Ruby install prefi
 "volatile".
 
 In addition to the [`Bootsnap::LoadPathCache::Cache`
-source](https://github.com/Shopify/bootsnap/blob/master/lib/bootsnap/load_path_cache/cache.rb),
+source](https://github.com/Shopify/bootsnap/blob/main/lib/bootsnap/load_path_cache/cache.rb),
 this diagram may help clarify how entry resolution works:
 
 ![How path searching works](https://cloud.githubusercontent.com/assets/3074765/25388270/670b5652-299b-11e7-87fb-975647f68981.png)
@@ -165,9 +192,9 @@ translated ruby source to an internal bytecode format, which is then executed by
 allows caching that bytecode. This allows us to bypass the relatively-expensive compilation step on
 subsequent loads of the same file.
 
-We also noticed that we spend a lot of time loading YAML documents during our application boot, and
-that MessagePack and Marshal are *much* faster at deserialization than YAML, even with a fast
-implementation. We use the same strategy of compilation caching for YAML documents, with the
+We also noticed that we spend a lot of time loading YAML and JSON documents during our application boot, and
+that MessagePack and Marshal are *much* faster at deserialization than YAML and JSON, even with a fast
+implementation. We use the same strategy of compilation caching for YAML and JSON documents, with the
 equivalent of Ruby's "bytecode" format being a MessagePack document (or, in the case of YAML
 documents with types unsupported by MessagePack, a Marshal stream).
 
@@ -214,9 +241,9 @@ Bootsnap writes a cache file containing a 64 byte header followed by the cache c
 is a cache key including several fields:
 
 * `version`, hardcoded in bootsnap. Essentially a schema version;
-* `os_version`, A hash of the current kernel version (on macOS, BSD) or glibc version (on Linux);
+* `ruby_platform`, A hash of `RUBY_PLATFORM` (e.g. x86_64-linux-gnu) variable.
 * `compile_option`, which changes with `RubyVM::InstructionSequence.compile_option` does;
-* `ruby_revision`, the version of Ruby this was compiled with;
+* `ruby_revision`, A hash of `RUBY_REVISION`, the exact version of Ruby;
 * `size`, the size of the source file;
 * `mtime`, the last-modification timestamp of the source file when it was compiled; and
 * `data_size`, the number of bytes following the header, which we need to read it into a buffer.
@@ -294,6 +321,19 @@ open    /c/nope.bundle -> -1
 # (nothing!)
 ```
 
+## Precompilation
+
+In development environments the bootsnap compilation cache is generated on the fly when source files are loaded.
+But in production environments, such as docker images, you might need to precompile the cache.
+
+To do so you can use the `bootsnap precompile` command.
+
+Example:
+
+```bash
+$ bundle exec bootsnap precompile --gemfile app/ lib/
+```
+
 ## When not to use Bootsnap
 
 *Alternative engines*: Bootsnap is pretty reliant on MRI features, and parts are disabled entirely on alternative ruby

data/exe/bootsnap

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "bootsnap/cli"
+exit Bootsnap::CLI.new(ARGV).run