bootsnap 0.2.2 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: fcd86efa96cf97561bb9f618b9be5a15394b764f
-  data.tar.gz: d6855e9fab0efbde45d0da337b2c6e63e4d272b1
+  metadata.gz: 3725f53309883dce83894e47eedbb4d8322eabd5
+  data.tar.gz: 81e14b1d48fc4431f6ac9bb2729cfb4a69aa0f5c
 SHA512:
-  metadata.gz: f4ad490b42d042304a9983fa0468ba202c5c494485e5e22215341c1b07b219a220fedec2c02b3f402a39b94a9eb953b2ba341e5c39fad4bc2681e5d9a5aa3445
-  data.tar.gz: 898ac7f4a00be925525bcee53ef03be8d9a2c022bff4afe9909f7dee339dd822c71b5c72b4c24a54605bd1c5f73ef342da5d82553b49a243c37351bd007db9f2
+  metadata.gz: 5a2b45cb55ceffdb5751561b3a35c476ea29bcd3d3a2a68a46521240d35cc1e61cf9ac90265db751b09967655969c7a685725878b5f80a61fd38a0efd7024458
+  data.tar.gz: 7f7975023cb1856d5ccada63e8be2e3ab94a751d4b80121821f2b01df6cbb94f5172041020ebb9300b63760da3fc2776e3b9b56160b95b4b0c07f82fbce07860

data/.travis.yml

@@ -0,0 +1,5 @@
+os: osx
+language: ruby
+rvm: ruby-2.4.0
+before_script: rake
+script: bin/testunit

data/CHANGELOG.md

@@ -0,0 +1,9 @@
+# 0.3.0
+
+* Migrate CompileCache from xattr as a cache backend to a cache directory
+    * Adds support for Linux and FreeBSD
+
+# 0.2.15
+
+* Support more versions of ActiveSupport (`depend_on`'s signature varies; don't reiterate it)
+* Fix bug in handling autoloaded modules that raise NoMethodError

data/README.md

@@ -1,8 +1,46 @@
-# Bootsnap
+# Bootsnap ![CI status](https://travis-ci.org/Shopify/bootsnap.svg?branch=master)
 
 **Beta-quality. See [the last section of this README](#trustworthiness).**
 
-Bootsnap is a library that plugs into a number of ruby and (optionally) `ActiveSupport` and `YAML`
+Bootsnap is a library that plugs into a number of Ruby and (optionally) `ActiveSupport` and `YAML`
+methods to optimize and cache expensive computations. See [the How Does This Work section](#how-does-this-work) for more information.
+
+#### Quick Performance Overview
+- [Discourse](https://github.com/discourse/discourse) reports a boot time reduction of approximately 50%, from roughly 6 to 3 seconds on one machine;
+- One of our smaller internal apps also sees a reduction of 50%, from 3.6 to 1.8 seconds;
+- The core Shopify platform -- a rather large monolithic application -- boots about 75% faster, dropping from around 25s to 6.5s.
+
+## Usage
+
+Add `bootsnap` to your `Gemfile`:
+
+```ruby
+gem 'bootsnap'
+```
+
+Next, add this to your boot setup immediately after `require 'bundler/setup'` (i.e. as early as
+possible: the sooner this is loaded, the sooner it can start optimizing things)
+
+```ruby
+require 'bootsnap'
+Bootsnap.setup(
+  cache_dir:            'tmp/cache', # Path to your cache
+  development_mode:     ENV['MY_ENV'] == 'development',
+  load_path_cache:      true,        # Should we optimize the LOAD_PATH with a cache?
+  autoload_paths_cache: true,        # Should we optimize ActiveSupport autoloads with cache?
+  disable_trace:        false,       # Sets `RubyVM::InstructionSequence.compile_option = { trace_instruction: false }`
+  compile_cache_iseq:   true,        # Should compile Ruby code into ISeq cache?
+  compile_cache_yaml:   true         # Should compile YAML into a cache?
+)
+```
+
+**Protip:** You can replace `require 'bootsnap'` with `BootLib::Require.from_gem('bootsnap',
+'bootsnap')` using [this trick](https://github.com/Shopify/bootsnap/wiki/Bootlib::Require). This
+will help optimize boot time further if you have an extremely large `$LOAD_PATH`.
+
+## How does this work?
+
+Bootsnap is a library that plugs into a number of Ruby and (optionally) `ActiveSupport` and `YAML`
 methods. These methods are modified to cache results of expensive computations, and can be grouped
 into two broad categories:
 
@@ -69,7 +107,8 @@ In addition to the [`Bootsnap::LoadPathCache::Cache`
 source](https://github.com/Shopify/bootsnap/blob/master/lib/bootsnap/load_path_cache/cache.rb),
 this diagram may help clarify how entry resolution works:
 
-![How path searching works](https://cloud.githubusercontent.com/assets/3074765/24532143/18278cd6-158c-11e7-8250-78d831df70db.png)
+![How path searching works](https://cloud.githubusercontent.com/assets/3074765/25388270/670b5652-299b-11e7-87fb-975647f68981.png)
+
 
 It's also important to note how expensive `LoadError`s can be. If ruby invokes
 `require 'something'`, but that file isn't on `$LOAD_PATH`, it takes `2 *
@@ -78,7 +117,8 @@ result too, raising a `LoadError` without touching the filesystem at all.
 
 ### Compilation Caching
 
-*(A simpler implementation of this concept can be found in [yomikomu](https://github.com/ko1/yomikomu)).*
+*(A more readable implementation of this concept can be found in
+[yomikomu](https://github.com/ko1/yomikomu)).*
 
 Ruby has complex grammar and parsing it is not a particularly cheap operation. Since 1.9, Ruby has
 translated ruby source to an internal bytecode format, which is then executed by the Ruby VM. Since
@@ -92,9 +132,8 @@ implementation. We use the same strategy of compilation caching for YAML documen
 equivalent of Ruby's "bytecode" format being a MessagePack document (or, in the case of YAML
 documents with types unsupported by MessagePack, a Marshal stream).
 
-These compilation results are stored using `xattr`s on the source files. This is likely to change in
-the future, as it has some limitations (notably precluding Linux support except where the user feels
-like changing mount flags). However, this is a very performant implementation. 
+These compilation results are stored in a cache directory, with filenames generated by taking a hash
+of the full expanded path of the input file (FNV1a-64).
 
 Whereas before, the sequence of syscalls generated to `require` a file would look like:
 
@@ -116,32 +155,36 @@ With bootsnap, we get:
 ```
 open      /c/foo.rb -> n
 fstat64   n
-fgetxattr n
-fgetxattr n
+close     n
+open      /c/foo.rb -> n
+fstat64   n
+open      (cache) -> m
+read      m
+read      m
+close     m
 close     n
 ```
 
-Bootsnap writes two `xattrs` attached to each file read:
+This may look worse at a glance, but underlies a large performance difference.
 
-* `user.aotcc.value`, the binary compilation result; and
-* `user.aotcc.key`, a cache key to determine whether `user.aotcc.value` is still valid.
+*(The first three syscalls in both listings -- `open`, `fstat64`, `close` -- are not inherently
+useful. [This ruby patch](https://bugs.ruby-lang.org/issues/13378) optimizes them out when coupled
+with bootsnap.)*
 
-The key includes several fields:
+Bootsnap writes a cache file containing a 64 byte header followed by the cache contents. The header
+is a cache key including several fields:
 
 * `version`, hardcoded in bootsnap. Essentially a schema version;
+* `os_version`, A hash of the current kernel version (on macOS, BSD) or glibc version (on Linux);
 * `compile_option`, which changes with `RubyVM::InstructionSequence.compile_option` does;
-* `data_size`, the number of bytes in `user.aotcc.value`, which we need to read it into a buffer
-  using `fgetxattr(2)`;
-* `ruby_revision`, the version of Ruby this was compiled with; and
-* `mtime`, the last-modification timestamp of the source file when it was compiled.
+* `ruby_revision`, the version of Ruby this was compiled with;
+* `size`, the size of the source file;
+* `mtime`, the last-modification timestamp of the source file when it was compiled; and
+* `data_size`, the number of bytes following the header, which we need to read it into a buffer.
 
 If the key is valid, the result is loaded from the value. Otherwise, it is regenerated and clobbers
 the current cache.
 
-This diagram may help illustrate how it works:
-
-![Compilation Caching](https://burkelibbey.s3.amazonaws.com/bootsnap-compile-cache.png)
-
 ### Putting it all together
 
 Imagine we have this file structure:
@@ -185,8 +228,13 @@ With bootsnap, we get:
 ```
 open      /c/foo.rb -> n
 fstat64   n
-fgetxattr n
-fgetxattr n
+close     n
+open      /c/foo.rb -> n
+fstat64   n
+open      (cache) -> m
+read      m
+read      m
+close     m
 close     n
 ```
 
@@ -207,40 +255,12 @@ open    /c/nope.bundle -> -1
 # (nothing!)
 ```
 
-## Usage
-
-Add `bootsnap` to your `Gemfile`:
-
-```ruby
-gem 'bootsnap'
-```
-
-Next, add this to your boot setup immediately after `require 'bundler/setup'` (i.e. as early as
-possible: the sooner this is loaded, the sooner it can start optimizing things)
-
-```ruby
-require 'bootsnap'
-Bootsnap.setup(
-  cache_dir:            'tmp/cache', # Path to your cache
-  development_mode:     ENV['MY_ENV'] == 'development',
-  load_path_cache:      true,        # Should we optimize the LOAD_PATH with a cache?
-  autoload_paths_cache: true,        # Should we optimize ActiveSupport autoloads with cache?
-  disable_trace:        false,       # Sets `RubyVM::InstructionSequence.compile_option = { trace_instruction: false }`
-  compile_cache_iseq:   true,        # Should compile Ruby code into ISeq cache?
-  compile_cache_yaml:   true         # Should compile YAML into a cache?
-)
-```
-
-**Protip:** You can replace `require 'bootsnap'` with `BootLib::Require.from_gem('bootsnap',
-'bootsnap')` using [this trick](https://github.com/Shopify/bootsnap/wiki/Bootlib::Require). This
-will help optimize boot time further if you have an extremely large `$LOAD_PATH`.
-
 ## Trustworthiness
 
 We use the `*_path_cache` features in production and haven't experienced any issues in a long time.
 
-The `compile_cache_*` features work well for us in development on macOS, but probably don't work on
-Linux at all.
+The `compile_cache_*` features work well for us in development on macOS. It should work on Linux,
+and we intend to deploy it in production, but haven't we haven't yet.
 
 `disable_trace` should be completely safe, but we don't really use it because some people like to
 use tools that make use of `trace` instructions.
@@ -250,5 +270,5 @@ use tools that make use of `trace` instructions.
 | `load_path_cache` | everywhere |
 | `autoload_path_cache` | everywhere |
 | `disable_trace` | nowhere, but it's safe unless you need tracing |
-| `compile_cache_iseq` | development, unlikely to work on Linux |
-| `compile_cache_yaml` | development, unlikely to work on Linux |
+| `compile_cache_iseq` | development, but probably safe to use everywhere |
+| `compile_cache_yaml` | development, but probably safe to use everywhere |

data/bootsnap.gemspec

@@ -23,12 +23,14 @@ Gem::Specification.new do |spec|
   spec.require_paths = ["lib"]
   spec.extensions    = ['ext/bootsnap/extconf.rb']
 
+  spec.required_ruby_version = '>= 2.3.0'
+
   spec.add_development_dependency "bundler", '~> 1'
   spec.add_development_dependency 'rake', '~> 10.0'
   spec.add_development_dependency 'rake-compiler', '~> 0'
   spec.add_development_dependency "minitest", "~> 5.0"
   spec.add_development_dependency "mocha", "~> 1.2"
+  spec.add_development_dependency "ffi-xattr", "~> 0.1.2"
 
   spec.add_runtime_dependency "msgpack", "~> 1.0"
-  spec.add_runtime_dependency "snappy", "~> 0.0.15"
 end

data/dev.yml

@@ -1,3 +1,6 @@
+env:
+  BOOTSNAP_PEDANTIC: '1'
+
 up:
   - ruby: 2.3.3
   - bundler