bootsnap 0.2.15 → 1.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: b0de1ff4a8bdd4478f422a07fba3981778828604
-  data.tar.gz: d9b7d5cc09e58be7bc31382be47d72870c60bf7f
+  metadata.gz: fbdf0cc19b09b396571d9d38cb037cebfcdab7b2
+  data.tar.gz: f07f07ec071dbffe3a48e1ae324457729a11ca61
 SHA512:
-  metadata.gz: f153d3e9591eb2fb8d2efecb958cbe21f61d93040f891ef63809285418cf76b3883d5490b3aa60d656bd066f96d7489317b13833bdf465058f7a339547177f6c
-  data.tar.gz: be2f217b2481c5d3eb44226d9861d2890dc7901143ab056e0d1f2fbd5e4f152ef497fb1cd67481ae44c4723dc5b0c93a6f6f6faebd705f4ce321aa0e5949bd2f
+  metadata.gz: 9db4ab87163f7c05a7b5c62fe17617149ec5acdc1caf186013c334222a4ad0a2087c5992b44fcf424201dd5b12448ce25826ccb4f99f495e450437aa90d0a0ae
+  data.tar.gz: 84714b27f6a3be1650807af151309dc789390944c1a317ec9c246067c2f4299184c182aa878c8eb5715da989c1c768a3d86992af02ab9d974f1d26d09bb2d170

data/CHANGELOG.md

@@ -1,3 +1,30 @@
+# 1.1.0
+
+* Add `bootsnap/setup`
+* Support jruby (without compile caching features)
+* Better deoptimization when Coverage is enabled
+* Consider `Bundler.bundle_path` to be stable
+
+# 1.0.0
+
+* (none)
+
+# 0.3.2
+
+* Minor performance savings around checking validity of cache in the presence of relative paths.
+* When coverage is enabled, skips optimization instead of exploding.
+
+# 0.3.1
+
+* Don't whitelist paths under `RbConfig::CONFIG['prefix']` as stable; instead use `['libdir']` (#41).
+* Catch `EOFError` when reading load-path-cache and regenerate cache.
+* Support relative paths in load-path-cache.
+
+# 0.3.0
+
+* Migrate CompileCache from xattr as a cache backend to a cache directory
+    * Adds support for Linux and FreeBSD
+
 # 0.2.15
 
 * Support more versions of ActiveSupport (`depend_on`'s signature varies; don't reiterate it)

data/README.md

@@ -1,39 +1,46 @@
-# Bootsnap ![CI status](https://travis-ci.org/Shopify/bootsnap.svg?branch=master)
+# Bootsnap [![Build Status](https://travis-ci.org/Shopify/bootsnap.svg?branch=master)](https://travis-ci.org/Shopify/bootsnap)
 
 **Beta-quality. See [the last section of this README](#trustworthiness).**
 
-**`compile_cache_*` features are only supported on MacOS (extremely likely to error on Linux;
-won't even compile on other platforms).**
+Bootsnap is a library that plugs into Ruby, with optional support for `ActiveSupport` and `YAML`,
+to optimize and cache expensive computations. See [How Does This Work](#how-does-this-work).
 
-Bootsnap is a library that plugs into a number of Ruby and (optionally) `ActiveSupport` and `YAML`
-methods to optimize and cache expensive computations. See [the How Does This Work section](#how-does-this-work) for more information.
-
-#### Quick Performance Overview
+#### Performance
 - [Discourse](https://github.com/discourse/discourse) reports a boot time reduction of approximately 50%, from roughly 6 to 3 seconds on one machine;
 - One of our smaller internal apps also sees a reduction of 50%, from 3.6 to 1.8 seconds;
 - The core Shopify platform -- a rather large monolithic application -- boots about 75% faster, dropping from around 25s to 6.5s.
 
 ## Usage
 
+This gem works on MacOS and Linux.
+
 Add `bootsnap` to your `Gemfile`:
 
 ```ruby
 gem 'bootsnap'
 ```
 
-Next, add this to your boot setup immediately after `require 'bundler/setup'` (i.e. as early as
-possible: the sooner this is loaded, the sooner it can start optimizing things)
+If you are using rails, add this to `config/boot.rb` immediately after `require 'bundler/setup'`:
+
+```ruby
+require 'bootsnap/setup'
+```
+
+If you are not using rails, or if you are but want more control over things, add this to your
+application setup immediately after `require 'bundler/setup'` (i.e. as early as possible: the sooner
+this is loaded, the sooner it can start optimizing things)
 
 ```ruby
 require 'bootsnap'
+env = ENV['RAILS_ENV'] || "development"
 Bootsnap.setup(
-  cache_dir:            'tmp/cache', # Path to your cache
-  development_mode:     ENV['MY_ENV'] == 'development',
-  load_path_cache:      true,        # Should we optimize the LOAD_PATH with a cache?
-  autoload_paths_cache: true,        # Should we optimize ActiveSupport autoloads with cache?
-  disable_trace:        false,       # Sets `RubyVM::InstructionSequence.compile_option = { trace_instruction: false }`
-  compile_cache_iseq:   true,        # Should compile Ruby code into ISeq cache?
-  compile_cache_yaml:   true         # Should compile YAML into a cache?
+  cache_dir:            'tmp/cache',          # Path to your cache
+  development_mode:     env == 'development', # Current working environment, e.g. RACK_ENV, RAILS_ENV, etc
+  load_path_cache:      true,                 # Optimize the LOAD_PATH with a cache
+  autoload_paths_cache: true,                 # Optimize ActiveSupport autoloads with cache
+  disable_trace:        true,                 # (Alpha) Set `RubyVM::InstructionSequence.compile_option = { trace_instruction: false }`
+  compile_cache_iseq:   true,                 # Compile Ruby code into ISeq cache, breaks coverage reporting.
+  compile_cache_yaml:   true                  # Compile YAML into a cache
 )
 ```
 
@@ -43,8 +50,7 @@ will help optimize boot time further if you have an extremely large `$LOAD_PATH`
 
 ## How does this work?
 
-Bootsnap is a library that plugs into a number of Ruby and (optionally) `ActiveSupport` and `YAML`
-methods. These methods are modified to cache results of expensive computations, and can be grouped
+Bootsnap optimizes methods to cache results of expensive computations, and can be grouped
 into two broad categories:
 
 * [Path Pre-Scanning](#path-pre-scanning)
@@ -104,7 +110,8 @@ entries do not expire -- once their contents has been scanned, it is assumed to
 
 The only directories considered "stable" are things under the Ruby install prefix
 (`RbConfig::CONFIG['prefix']`, e.g. `/usr/local/ruby` or `~/.rubies/x.y.z`), and things under the
-`Gem.path` (e.g. `~/.gem/ruby/x.y.z`). Everything else is considered "volatile".
+`Gem.path` (e.g. `~/.gem/ruby/x.y.z`) or `Bundler.bundle_path`. Everything else is considered
+"volatile".
 
 In addition to the [`Bootsnap::LoadPathCache::Cache`
 source](https://github.com/Shopify/bootsnap/blob/master/lib/bootsnap/load_path_cache/cache.rb),
@@ -120,7 +127,8 @@ result too, raising a `LoadError` without touching the filesystem at all.
 
 ### Compilation Caching
 
-*(A simpler implementation of this concept can be found in [yomikomu](https://github.com/ko1/yomikomu)).*
+*(A more readable implementation of this concept can be found in
+[yomikomu](https://github.com/ko1/yomikomu)).*
 
 Ruby has complex grammar and parsing it is not a particularly cheap operation. Since 1.9, Ruby has
 translated ruby source to an internal bytecode format, which is then executed by the Ruby VM. Since
@@ -134,9 +142,8 @@ implementation. We use the same strategy of compilation caching for YAML documen
 equivalent of Ruby's "bytecode" format being a MessagePack document (or, in the case of YAML
 documents with types unsupported by MessagePack, a Marshal stream).
 
-These compilation results are stored using `xattr`s on the source files. This is likely to change in
-the future, as it has some limitations (notably precluding Linux support except where the user feels
-like changing mount flags). However, this is a very performant implementation. 
+These compilation results are stored in a cache directory, with filenames generated by taking a hash
+of the full expanded path of the input file (FNV1a-64).
 
 Whereas before, the sequence of syscalls generated to `require` a file would look like:
 
@@ -158,32 +165,36 @@ With bootsnap, we get:
 ```
 open      /c/foo.rb -> n
 fstat64   n
-fgetxattr n
-fgetxattr n
+close     n
+open      /c/foo.rb -> n
+fstat64   n
+open      (cache) -> m
+read      m
+read      m
+close     m
 close     n
 ```
 
-Bootsnap writes two `xattrs` attached to each file read:
+This may look worse at a glance, but underlies a large performance difference.
 
-* `user.aotcc.value`, the binary compilation result; and
-* `user.aotcc.key`, a cache key to determine whether `user.aotcc.value` is still valid.
+*(The first three syscalls in both listings -- `open`, `fstat64`, `close` -- are not inherently
+useful. [This ruby patch](https://bugs.ruby-lang.org/issues/13378) optimizes them out when coupled
+with bootsnap.)*
 
-The key includes several fields:
+Bootsnap writes a cache file containing a 64 byte header followed by the cache contents. The header
+is a cache key including several fields:
 
 * `version`, hardcoded in bootsnap. Essentially a schema version;
+* `os_version`, A hash of the current kernel version (on macOS, BSD) or glibc version (on Linux);
 * `compile_option`, which changes with `RubyVM::InstructionSequence.compile_option` does;
-* `data_size`, the number of bytes in `user.aotcc.value`, which we need to read it into a buffer
-  using `fgetxattr(2)`;
-* `ruby_revision`, the version of Ruby this was compiled with; and
-* `mtime`, the last-modification timestamp of the source file when it was compiled.
+* `ruby_revision`, the version of Ruby this was compiled with;
+* `size`, the size of the source file;
+* `mtime`, the last-modification timestamp of the source file when it was compiled; and
+* `data_size`, the number of bytes following the header, which we need to read it into a buffer.
 
 If the key is valid, the result is loaded from the value. Otherwise, it is regenerated and clobbers
 the current cache.
 
-This diagram may help illustrate how it works:
-
-![Compilation Caching](https://burkelibbey.s3.amazonaws.com/bootsnap-compile-cache.png)
-
 ### Putting it all together
 
 Imagine we have this file structure:
@@ -227,8 +238,13 @@ With bootsnap, we get:
 ```
 open      /c/foo.rb -> n
 fstat64   n
-fgetxattr n
-fgetxattr n
+close     n
+open      /c/foo.rb -> n
+fstat64   n
+open      (cache) -> m
+read      m
+read      m
+close     m
 close     n
 ```
 
@@ -253,8 +269,8 @@ open    /c/nope.bundle -> -1
 
 We use the `*_path_cache` features in production and haven't experienced any issues in a long time.
 
-The `compile_cache_*` features work well for us in development on macOS, but probably don't work on
-Linux at all.
+The `compile_cache_*` features work well for us in development on macOS. It should work on Linux,
+and we intend to deploy it in production, but we haven't yet.
 
 `disable_trace` should be completely safe, but we don't really use it because some people like to
 use tools that make use of `trace` instructions.
@@ -264,5 +280,5 @@ use tools that make use of `trace` instructions.
 | `load_path_cache` | everywhere |
 | `autoload_path_cache` | everywhere |
 | `disable_trace` | nowhere, but it's safe unless you need tracing |
-| `compile_cache_iseq` | development, unlikely to work on Linux |
-| `compile_cache_yaml` | development, unlikely to work on Linux |
+| `compile_cache_iseq` | development, but probably safe to use everywhere |
+| `compile_cache_yaml` | development, but probably safe to use everywhere |

data/bootsnap.gemspec

@@ -18,19 +18,22 @@ Gem::Specification.new do |spec|
   spec.files = `git ls-files -z`.split("\x0").reject do |f|
     f.match(%r{^(test|spec|features)/})
   end
-  spec.bindir        = "exe"
-  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
   spec.require_paths = ["lib"]
-  spec.extensions    = ['ext/bootsnap/extconf.rb']
 
-  spec.required_ruby_version = '>= 2.3.0'
+  spec.required_ruby_version = '>= 2.0.0'
+
+  if RUBY_PLATFORM =~ /java/
+    spec.platform = 'java'
+  else
+    spec.platform    = Gem::Platform::RUBY
+    spec.extensions  = ['ext/bootsnap/extconf.rb']
+  end
 
   spec.add_development_dependency "bundler", '~> 1'
   spec.add_development_dependency 'rake', '~> 10.0'
   spec.add_development_dependency 'rake-compiler', '~> 0'
   spec.add_development_dependency "minitest", "~> 5.0"
   spec.add_development_dependency "mocha", "~> 1.2"
-  spec.add_development_dependency "ffi-xattr", "~> 0.1.2"
 
   spec.add_runtime_dependency "msgpack", "~> 1.0"
 end