bootsnap 1.8.1 → 1.10.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +78 -3
- data/LICENSE.txt +1 -1
- data/README.md +2 -2
- data/exe/bootsnap +1 -1
- data/ext/bootsnap/bootsnap.c +39 -37
- data/ext/bootsnap/extconf.rb +13 -11
- data/lib/bootsnap/bundler.rb +1 -0
- data/lib/bootsnap/cli/worker_pool.rb +1 -0
- data/lib/bootsnap/cli.rb +78 -43
- data/lib/bootsnap/compile_cache/iseq.rb +42 -12
- data/lib/bootsnap/compile_cache/json.rb +88 -0
- data/lib/bootsnap/compile_cache/yaml.rb +264 -77
- data/lib/bootsnap/compile_cache.rb +22 -7
- data/lib/bootsnap/explicit_require.rb +4 -3
- data/lib/bootsnap/load_path_cache/cache.rb +57 -41
- data/lib/bootsnap/load_path_cache/change_observer.rb +2 -0
- data/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb +33 -65
- data/lib/bootsnap/load_path_cache/core_ext/loaded_features.rb +1 -0
- data/lib/bootsnap/load_path_cache/loaded_features_index.rb +32 -21
- data/lib/bootsnap/load_path_cache/path.rb +5 -3
- data/lib/bootsnap/load_path_cache/path_scanner.rb +6 -5
- data/lib/bootsnap/load_path_cache/realpath_cache.rb +1 -0
- data/lib/bootsnap/load_path_cache/store.rb +24 -7
- data/lib/bootsnap/load_path_cache.rb +16 -22
- data/lib/bootsnap/setup.rb +2 -1
- data/lib/bootsnap/version.rb +2 -1
- data/lib/bootsnap.rb +103 -89
- metadata +6 -75
    
        checksums.yaml
    CHANGED
    
    | @@ -1,7 +1,7 @@ | |
| 1 1 | 
             
            ---
         | 
| 2 2 | 
             
            SHA256:
         | 
| 3 | 
            -
              metadata.gz:  | 
| 4 | 
            -
              data.tar.gz:  | 
| 3 | 
            +
              metadata.gz: 3dfd9cfb2cde26fc31c82206b73f3940b4b46903a3fa73d0f3f1a4c4a90c6984
         | 
| 4 | 
            +
              data.tar.gz: 6dd8acfb2676a2585eb320ed50ba0ddb9f9c847fb415bcda824c7ebff0cce752
         | 
| 5 5 | 
             
            SHA512:
         | 
| 6 | 
            -
              metadata.gz:  | 
| 7 | 
            -
              data.tar.gz:  | 
| 6 | 
            +
              metadata.gz: 219a84be718e4ac5681621739b6f9400d77a16dee631bdc286adad87c0ddf99cc581da7281150add34075c08c59832df2a468524656d2f1ac36ac1613be051bb
         | 
| 7 | 
            +
              data.tar.gz: 0e407a034fd081c9e506637e0997d0c2cc0eea312be798f6fd0d573173699f906f1b43a1f3d37ff64ee6efb2f9257228c6c4e7663b7337bfc49d99e5089fccb9
         | 
    
        data/CHANGELOG.md
    CHANGED
    
    | @@ -1,12 +1,87 @@ | |
| 1 1 | 
             
            # Unreleased
         | 
| 2 2 |  | 
| 3 | 
            +
            # 1.10.3
         | 
| 4 | 
            +
             | 
| 5 | 
            +
            * Fix Regexp and Date type support in YAML compile cache. (#400)
         | 
| 6 | 
            +
             | 
| 7 | 
            +
            * Improve the YAML compile cache to support `UTF-8` symbols. (#398, #399)
         | 
| 8 | 
            +
              [The default `MessagePack` symbol serializer assumes all symbols are ASCII](https://github.com/msgpack/msgpack-ruby/pull/211),
         | 
| 9 | 
            +
              because of this, non-ASCII compatible symbol would be restored with `ASCII_8BIT` encoding (AKA `BINARY`).
         | 
| 10 | 
            +
              Bootsnap now properly cache them in `UTF-8`.
         | 
| 11 | 
            +
             | 
| 12 | 
            +
              Note that the above only apply for actual YAML symbols (e..g `--- :foo`).
         | 
| 13 | 
            +
              The issue is still present for string keys parsed with `YAML.load_file(..., symbolize_names: true)`, that is a bug
         | 
| 14 | 
            +
              in `msgpack` that will hopefully be solved soon, see: https://github.com/msgpack/msgpack-ruby/pull/246
         | 
| 15 | 
            +
             | 
| 16 | 
            +
            * Entirely disable the YAML compile cache if `Encoding.default_internal` is set to an encoding not supported by `msgpack`. (#398)
         | 
| 17 | 
            +
              `Psych` coerce strings to `Encoding.default_internal`, but `MessagePack` doesn't. So in this scenario we can't provide
         | 
| 18 | 
            +
              YAML caching at all without returning the strings in the wrong encoding.
         | 
| 19 | 
            +
              This never came up in practice but might as well be safe.
         | 
| 20 | 
            +
             | 
| 21 | 
            +
            # 1.10.2
         | 
| 22 | 
            +
             | 
| 23 | 
            +
            * Reduce the `Kernel.require` extra stack frames some more. Now bootsnap should only add one extra frame per `require` call.
         | 
| 24 | 
            +
             | 
| 25 | 
            +
            * Better check `freeze` option support in JSON compile cache.
         | 
| 26 | 
            +
              Previously `JSON.load_file(..., freeze: true)` would be cached even when the msgpack version is missing support for it.
         | 
| 27 | 
            +
             | 
| 28 | 
            +
            # 1.10.1
         | 
| 29 | 
            +
             | 
| 30 | 
            +
            * Fix `Kernel#autoload`'s fallback path always being executed.
         | 
| 31 | 
            +
             | 
| 32 | 
            +
            * Consider `unlink` failing with `ENOENT` as a success.
         | 
| 33 | 
            +
             | 
| 34 | 
            +
            # 1.10.0
         | 
| 35 | 
            +
             | 
| 36 | 
            +
            * Delay requiring `FileUtils`. (#285)
         | 
| 37 | 
            +
              `FileUtils` can be installed as a gem, so it's best to wait for bundler to have setup the load path before requiring it.
         | 
| 38 | 
            +
             | 
| 39 | 
            +
            * Improve support of Psych 4. (#392)
         | 
| 40 | 
            +
              Since `1.8.0`, `YAML.load_file` was no longer cached when Psych 4 was used. This is because `load_file` loads
         | 
| 41 | 
            +
              in safe mode by default, so the Bootsnap cache could defeat that safety.
         | 
| 42 | 
            +
              Now when precompiling YAML files, Bootsnap first try to parse them in safe mode, and if it can't fallback to unsafe mode,
         | 
| 43 | 
            +
              and the cache contains a flag that records whether it was generated in safe mode or not.
         | 
| 44 | 
            +
              `YAML.unsafe_load_file` will use safe caches just fine, but `YAML.load_file` will fallback to uncached YAML parsing
         | 
| 45 | 
            +
              if the cache was generated using unsafe parsing.
         | 
| 46 | 
            +
             | 
| 47 | 
            +
            * Minimize the Kernel.require extra stack frames. (#393)
         | 
| 48 | 
            +
              This should reduce the noise generated by bootsnap on `LoadError`.
         | 
| 49 | 
            +
             | 
| 50 | 
            +
            # 1.9.4
         | 
| 51 | 
            +
             | 
| 52 | 
            +
            * Ignore absolute paths in the loaded feature index. (#385)
         | 
| 53 | 
            +
              This fixes a compatibility issue with Zeitwerk when Zeitwerk is loaded before bootsnap. It also should
         | 
| 54 | 
            +
              reduce the memory usage and improve load performance of Zeitwerk managed files.
         | 
| 55 | 
            +
             | 
| 56 | 
            +
            * Automatically invalidate the load path cache whenever the Ruby version change. (#387)
         | 
| 57 | 
            +
              This is to avoid issues in case the same installation path is re-used for subsequent ruby patch releases.
         | 
| 58 | 
            +
             | 
| 59 | 
            +
            # 1.9.3
         | 
| 60 | 
            +
             | 
| 61 | 
            +
            * Only disable the compile cache for source files impacted by [Ruby 3.0.3 [Bug 18250]](https://bugs.ruby-lang.org/issues/18250).
         | 
| 62 | 
            +
              This should keep the performance loss to a minimum.
         | 
| 63 | 
            +
             | 
| 64 | 
            +
            # 1.9.2
         | 
| 65 | 
            +
             | 
| 66 | 
            +
            * Disable compile cache if [Ruby 3.0.3's ISeq cache bug](https://bugs.ruby-lang.org/issues/18250) is detected.
         | 
| 67 | 
            +
              AKA `iseq.rb:13 to_binary: wrong argument type false (expected Symbol)`
         | 
| 68 | 
            +
            * Fix `Kernel.load` behavior: before `load 'a'` would load `a.rb` (and other tried extensions) and wouldn't load `a` unless `development_mode: true`, now only `a` would be loaded and files with extensions wouldn't be.
         | 
| 69 | 
            +
             | 
| 70 | 
            +
            # 1.9.1
         | 
| 71 | 
            +
             | 
| 72 | 
            +
            * Removed a forgotten debug statement in JSON precompilation.
         | 
| 73 | 
            +
             | 
| 74 | 
            +
            # 1.9.0
         | 
| 75 | 
            +
             | 
| 76 | 
            +
            * Added a compilation cache for `JSON.load_file`. (#370)
         | 
| 77 | 
            +
             | 
| 3 78 | 
             
            # 1.8.1
         | 
| 4 79 |  | 
| 5 80 | 
             
            * Fixed support for older Psych. (#369)
         | 
| 6 81 |  | 
| 7 82 | 
             
            # 1.8.0
         | 
| 8 83 |  | 
| 9 | 
            -
            * Improve support for  | 
| 84 | 
            +
            * Improve support for Psych 4. (#368)
         | 
| 10 85 |  | 
| 11 86 | 
             
            # 1.7.7
         | 
| 12 87 |  | 
| @@ -24,8 +99,8 @@ | |
| 24 99 |  | 
| 25 100 | 
             
            # 1.7.4
         | 
| 26 101 |  | 
| 27 | 
            -
            * Stop raising errors when  | 
| 28 | 
            -
              if somehow it can't be saved,  | 
| 102 | 
            +
            * Stop raising errors when encountering various file system errors. The cache is now best effort,
         | 
| 103 | 
            +
              if somehow it can't be saved, bootsnap will gracefully fallback to the original operation (e.g. `Kernel.require`).
         | 
| 29 104 | 
             
              (#353, #177, #262)
         | 
| 30 105 |  | 
| 31 106 | 
             
            # 1.7.3
         | 
    
        data/LICENSE.txt
    CHANGED
    
    
    
        data/README.md
    CHANGED
    
    | @@ -41,7 +41,7 @@ getting progressively slower, this is almost certainly the cause.** | |
| 41 41 | 
             
            It's technically possible to simply specify `gem 'bootsnap', require: 'bootsnap/setup'`, but it's
         | 
| 42 42 | 
             
            important to load Bootsnap as early as possible to get maximum performance improvement.
         | 
| 43 43 |  | 
| 44 | 
            -
            You can see how this require works [here](https://github.com/Shopify/bootsnap/blob/ | 
| 44 | 
            +
            You can see how this require works [here](https://github.com/Shopify/bootsnap/blob/main/lib/bootsnap/setup.rb).
         | 
| 45 45 |  | 
| 46 46 | 
             
            If you are not using Rails, or if you are but want more control over things, add this to your
         | 
| 47 47 | 
             
            application setup immediately after `require 'bundler/setup'` (i.e. as early as possible: the sooner
         | 
| @@ -161,7 +161,7 @@ The only directories considered "stable" are things under the Ruby install prefi | |
| 161 161 | 
             
            "volatile".
         | 
| 162 162 |  | 
| 163 163 | 
             
            In addition to the [`Bootsnap::LoadPathCache::Cache`
         | 
| 164 | 
            -
            source](https://github.com/Shopify/bootsnap/blob/ | 
| 164 | 
            +
            source](https://github.com/Shopify/bootsnap/blob/main/lib/bootsnap/load_path_cache/cache.rb),
         | 
| 165 165 | 
             
            this diagram may help clarify how entry resolution works:
         | 
| 166 166 |  | 
| 167 167 | 
             
            
         | 
    
        data/exe/bootsnap
    CHANGED
    
    
    
        data/ext/bootsnap/bootsnap.c
    CHANGED
    
    | @@ -2,7 +2,7 @@ | |
| 2 2 | 
             
             * Suggested reading order:
         | 
| 3 3 | 
             
             * 1. Skim Init_bootsnap
         | 
| 4 4 | 
             
             * 2. Skim bs_fetch
         | 
| 5 | 
            -
             * 3. The rest of  | 
| 5 | 
            +
             * 3. The rest of everyrything
         | 
| 6 6 | 
             
             *
         | 
| 7 7 | 
             
             * Init_bootsnap sets up the ruby objects and binds bs_fetch to
         | 
| 8 8 | 
             
             * Bootsnap::CompileCache::Native.fetch.
         | 
| @@ -75,7 +75,7 @@ struct bs_cache_key { | |
| 75 75 | 
             
            STATIC_ASSERT(sizeof(struct bs_cache_key) == KEY_SIZE);
         | 
| 76 76 |  | 
| 77 77 | 
             
            /* Effectively a schema version. Bumping invalidates all previous caches */
         | 
| 78 | 
            -
            static const uint32_t current_version =  | 
| 78 | 
            +
            static const uint32_t current_version = 4;
         | 
| 79 79 |  | 
| 80 80 | 
             
            /* hash of e.g. "x86_64-darwin17", invalidating when ruby is recompiled on a
         | 
| 81 81 | 
             
             * new OS ABI, etc. */
         | 
| @@ -91,8 +91,7 @@ static mode_t current_umask; | |
| 91 91 | 
             
            static VALUE rb_mBootsnap;
         | 
| 92 92 | 
             
            static VALUE rb_mBootsnap_CompileCache;
         | 
| 93 93 | 
             
            static VALUE rb_mBootsnap_CompileCache_Native;
         | 
| 94 | 
            -
            static VALUE  | 
| 95 | 
            -
            static ID uncompilable;
         | 
| 94 | 
            +
            static VALUE rb_cBootsnap_CompileCache_UNCOMPILABLE;
         | 
| 96 95 | 
             
            static ID instrumentation_method;
         | 
| 97 96 | 
             
            static VALUE sym_miss;
         | 
| 98 97 | 
             
            static VALUE sym_stale;
         | 
| @@ -120,10 +119,8 @@ static uint32_t get_ruby_platform(void); | |
| 120 119 | 
             
             * exception.
         | 
| 121 120 | 
             
             */
         | 
| 122 121 | 
             
            static int bs_storage_to_output(VALUE handler, VALUE args, VALUE storage_data, VALUE * output_data);
         | 
| 123 | 
            -
            static VALUE prot_storage_to_output(VALUE arg);
         | 
| 124 122 | 
             
            static VALUE prot_input_to_output(VALUE arg);
         | 
| 125 123 | 
             
            static void bs_input_to_output(VALUE handler, VALUE args, VALUE input_data, VALUE * output_data, int * exception_tag);
         | 
| 126 | 
            -
            static VALUE prot_input_to_storage(VALUE arg);
         | 
| 127 124 | 
             
            static int bs_input_to_storage(VALUE handler, VALUE args, VALUE input_data, VALUE pathval, VALUE * storage_data);
         | 
| 128 125 | 
             
            struct s2o_data;
         | 
| 129 126 | 
             
            struct i2o_data;
         | 
| @@ -151,12 +148,12 @@ Init_bootsnap(void) | |
| 151 148 | 
             
              rb_mBootsnap = rb_define_module("Bootsnap");
         | 
| 152 149 | 
             
              rb_mBootsnap_CompileCache = rb_define_module_under(rb_mBootsnap, "CompileCache");
         | 
| 153 150 | 
             
              rb_mBootsnap_CompileCache_Native = rb_define_module_under(rb_mBootsnap_CompileCache, "Native");
         | 
| 154 | 
            -
               | 
| 151 | 
            +
              rb_cBootsnap_CompileCache_UNCOMPILABLE = rb_const_get(rb_mBootsnap_CompileCache, rb_intern("UNCOMPILABLE"));
         | 
| 152 | 
            +
              rb_global_variable(&rb_cBootsnap_CompileCache_UNCOMPILABLE);
         | 
| 155 153 |  | 
| 156 154 | 
             
              current_ruby_revision = get_ruby_revision();
         | 
| 157 155 | 
             
              current_ruby_platform = get_ruby_platform();
         | 
| 158 156 |  | 
| 159 | 
            -
              uncompilable = rb_intern("__bootsnap_uncompilable__");
         | 
| 160 157 | 
             
              instrumentation_method = rb_intern("_instrument");
         | 
| 161 158 |  | 
| 162 159 | 
             
              sym_miss = ID2SYM(rb_intern("miss"));
         | 
| @@ -426,6 +423,7 @@ open_current_file(char * path, struct bs_cache_key * key, const char ** errno_pr | |
| 426 423 | 
             
            #define ERROR_WITH_ERRNO -1
         | 
| 427 424 | 
             
            #define CACHE_MISS -2
         | 
| 428 425 | 
             
            #define CACHE_STALE -3
         | 
| 426 | 
            +
            #define CACHE_UNCOMPILABLE -4
         | 
| 429 427 |  | 
| 430 428 | 
             
            /*
         | 
| 431 429 | 
             
             * Read the cache key from the given fd, which must have position 0 (e.g.
         | 
| @@ -507,14 +505,14 @@ fetch_cached_data(int fd, ssize_t data_size, VALUE handler, VALUE args, VALUE * | |
| 507 505 | 
             
              if (data_size > 100000000000) {
         | 
| 508 506 | 
             
                *errno_provenance = "bs_fetch:fetch_cached_data:datasize";
         | 
| 509 507 | 
             
                errno = EINVAL; /* because wtf? */
         | 
| 510 | 
            -
                ret =  | 
| 508 | 
            +
                ret = ERROR_WITH_ERRNO;
         | 
| 511 509 | 
             
                goto done;
         | 
| 512 510 | 
             
              }
         | 
| 513 511 | 
             
              data = ALLOC_N(char, data_size);
         | 
| 514 512 | 
             
              nread = read(fd, data, data_size);
         | 
| 515 513 | 
             
              if (nread < 0) {
         | 
| 516 514 | 
             
                *errno_provenance = "bs_fetch:fetch_cached_data:read";
         | 
| 517 | 
            -
                ret =  | 
| 515 | 
            +
                ret = ERROR_WITH_ERRNO;
         | 
| 518 516 | 
             
                goto done;
         | 
| 519 517 | 
             
              }
         | 
| 520 518 | 
             
              if (nread != data_size) {
         | 
| @@ -525,6 +523,10 @@ fetch_cached_data(int fd, ssize_t data_size, VALUE handler, VALUE args, VALUE * | |
| 525 523 | 
             
              storage_data = rb_str_new(data, data_size);
         | 
| 526 524 |  | 
| 527 525 | 
             
              *exception_tag = bs_storage_to_output(handler, args, storage_data, output_data);
         | 
| 526 | 
            +
              if (*output_data == rb_cBootsnap_CompileCache_UNCOMPILABLE) {
         | 
| 527 | 
            +
                ret = CACHE_UNCOMPILABLE;
         | 
| 528 | 
            +
                goto done;
         | 
| 529 | 
            +
              }
         | 
| 528 530 | 
             
              ret = 0;
         | 
| 529 531 | 
             
            done:
         | 
| 530 532 | 
             
              if (data != NULL) xfree(data);
         | 
| @@ -737,7 +739,15 @@ bs_fetch(char * path, VALUE path_v, char * cache_path, VALUE handler, VALUE args | |
| 737 739 | 
             
                  &output_data, &exception_tag, &errno_provenance
         | 
| 738 740 | 
             
                );
         | 
| 739 741 | 
             
                if (exception_tag != 0) goto raise;
         | 
| 740 | 
            -
                else if (res ==  | 
| 742 | 
            +
                else if (res == CACHE_UNCOMPILABLE) {
         | 
| 743 | 
            +
                  /* If fetch_cached_data returned `Uncompilable` we fallback to `input_to_output`
         | 
| 744 | 
            +
                    This happens if we have say, an unsafe YAML cache, but try to load it in safe mode */
         | 
| 745 | 
            +
                  if (bs_read_contents(current_fd, current_key.size, &contents, &errno_provenance) < 0) goto fail_errno;
         | 
| 746 | 
            +
                  input_data = rb_str_new(contents, current_key.size);
         | 
| 747 | 
            +
                  bs_input_to_output(handler, args, input_data, &output_data, &exception_tag);
         | 
| 748 | 
            +
                  if (exception_tag != 0) goto raise;
         | 
| 749 | 
            +
                  goto succeed;
         | 
| 750 | 
            +
                } else if (res == CACHE_MISS || res == CACHE_STALE) valid_cache = 0;
         | 
| 741 751 | 
             
                else if (res == ERROR_WITH_ERRNO) goto fail_errno;
         | 
| 742 752 | 
             
                else if (!NIL_P(output_data)) goto succeed; /* fast-path, goal */
         | 
| 743 753 | 
             
              }
         | 
| @@ -754,7 +764,7 @@ bs_fetch(char * path, VALUE path_v, char * cache_path, VALUE handler, VALUE args | |
| 754 764 | 
             
              if (exception_tag != 0) goto raise;
         | 
| 755 765 | 
             
              /* If input_to_storage raised Bootsnap::CompileCache::Uncompilable, don't try
         | 
| 756 766 | 
             
               * to cache anything; just return input_to_output(input_data) */
         | 
| 757 | 
            -
              if (storage_data ==  | 
| 767 | 
            +
              if (storage_data == rb_cBootsnap_CompileCache_UNCOMPILABLE) {
         | 
| 758 768 | 
             
                bs_input_to_output(handler, args, input_data, &output_data, &exception_tag);
         | 
| 759 769 | 
             
                if (exception_tag != 0) goto raise;
         | 
| 760 770 | 
             
                goto succeed;
         | 
| @@ -772,12 +782,20 @@ bs_fetch(char * path, VALUE path_v, char * cache_path, VALUE handler, VALUE args | |
| 772 782 | 
             
              exception_tag = bs_storage_to_output(handler, args, storage_data, &output_data);
         | 
| 773 783 | 
             
              if (exception_tag != 0) goto raise;
         | 
| 774 784 |  | 
| 775 | 
            -
               | 
| 776 | 
            -
             | 
| 777 | 
            -
             | 
| 785 | 
            +
              if (output_data == rb_cBootsnap_CompileCache_UNCOMPILABLE) {
         | 
| 786 | 
            +
                /* If storage_to_output returned `Uncompilable` we fallback to `input_to_output` */
         | 
| 787 | 
            +
                bs_input_to_output(handler, args, input_data, &output_data, &exception_tag);
         | 
| 788 | 
            +
                if (exception_tag != 0) goto raise;
         | 
| 789 | 
            +
              } else if (NIL_P(output_data)) {
         | 
| 790 | 
            +
                /* If output_data is nil, delete the cache entry and generate the output
         | 
| 791 | 
            +
                 * using input_to_output */
         | 
| 778 792 | 
             
                if (unlink(cache_path) < 0) {
         | 
| 779 | 
            -
                   | 
| 780 | 
            -
                   | 
| 793 | 
            +
                  /* If the cache was already deleted, it might be that another process did it before us.
         | 
| 794 | 
            +
                  * No point raising an error */
         | 
| 795 | 
            +
                  if (errno != ENOENT) {
         | 
| 796 | 
            +
                    errno_provenance = "bs_fetch:unlink";
         | 
| 797 | 
            +
                    goto fail_errno;
         | 
| 798 | 
            +
                  }
         | 
| 781 799 | 
             
                }
         | 
| 782 800 | 
             
                bs_input_to_output(handler, args, input_data, &output_data, &exception_tag);
         | 
| 783 801 | 
             
                if (exception_tag != 0) goto raise;
         | 
| @@ -856,7 +874,7 @@ bs_precompile(char * path, VALUE path_v, char * cache_path, VALUE handler) | |
| 856 874 |  | 
| 857 875 | 
             
              /* If input_to_storage raised Bootsnap::CompileCache::Uncompilable, don't try
         | 
| 858 876 | 
             
               * to cache anything; just return false */
         | 
| 859 | 
            -
              if (storage_data ==  | 
| 877 | 
            +
              if (storage_data == rb_cBootsnap_CompileCache_UNCOMPILABLE) {
         | 
| 860 878 | 
             
                goto fail;
         | 
| 861 879 | 
             
              }
         | 
| 862 880 | 
             
              /* If storage_data isn't a string, we can't cache it */
         | 
| @@ -919,7 +937,7 @@ struct i2s_data { | |
| 919 937 | 
             
            };
         | 
| 920 938 |  | 
| 921 939 | 
             
            static VALUE
         | 
| 922 | 
            -
             | 
| 940 | 
            +
            try_storage_to_output(VALUE arg)
         | 
| 923 941 | 
             
            {
         | 
| 924 942 | 
             
              struct s2o_data * data = (struct s2o_data *)arg;
         | 
| 925 943 | 
             
              return rb_funcall(data->handler, rb_intern("storage_to_output"), 2, data->storage_data, data->args);
         | 
| @@ -934,7 +952,7 @@ bs_storage_to_output(VALUE handler, VALUE args, VALUE storage_data, VALUE * outp | |
| 934 952 | 
             
                .args         = args,
         | 
| 935 953 | 
             
                .storage_data = storage_data,
         | 
| 936 954 | 
             
              };
         | 
| 937 | 
            -
              *output_data = rb_protect( | 
| 955 | 
            +
              *output_data = rb_protect(try_storage_to_output, (VALUE)&s2o_data, &state);
         | 
| 938 956 | 
             
              return state;
         | 
| 939 957 | 
             
            }
         | 
| 940 958 |  | 
| @@ -963,22 +981,6 @@ try_input_to_storage(VALUE arg) | |
| 963 981 | 
             
              return rb_funcall(data->handler, rb_intern("input_to_storage"), 2, data->input_data, data->pathval);
         | 
| 964 982 | 
             
            }
         | 
| 965 983 |  | 
| 966 | 
            -
            static VALUE
         | 
| 967 | 
            -
            rescue_input_to_storage(VALUE arg, VALUE e)
         | 
| 968 | 
            -
            {
         | 
| 969 | 
            -
              return uncompilable;
         | 
| 970 | 
            -
            }
         | 
| 971 | 
            -
             | 
| 972 | 
            -
            static VALUE
         | 
| 973 | 
            -
            prot_input_to_storage(VALUE arg)
         | 
| 974 | 
            -
            {
         | 
| 975 | 
            -
              struct i2s_data * data = (struct i2s_data *)arg;
         | 
| 976 | 
            -
              return rb_rescue2(
         | 
| 977 | 
            -
                  try_input_to_storage, (VALUE)data,
         | 
| 978 | 
            -
                  rescue_input_to_storage, Qnil,
         | 
| 979 | 
            -
                  rb_eBootsnap_CompileCache_Uncompilable, 0);
         | 
| 980 | 
            -
            }
         | 
| 981 | 
            -
             | 
| 982 984 | 
             
            static int
         | 
| 983 985 | 
             
            bs_input_to_storage(VALUE handler, VALUE args, VALUE input_data, VALUE pathval, VALUE * storage_data)
         | 
| 984 986 | 
             
            {
         | 
| @@ -988,6 +990,6 @@ bs_input_to_storage(VALUE handler, VALUE args, VALUE input_data, VALUE pathval, | |
| 988 990 | 
             
                .input_data = input_data,
         | 
| 989 991 | 
             
                .pathval    = pathval,
         | 
| 990 992 | 
             
              };
         | 
| 991 | 
            -
              *storage_data = rb_protect( | 
| 993 | 
            +
              *storage_data = rb_protect(try_input_to_storage, (VALUE)&i2s_data, &state);
         | 
| 992 994 | 
             
              return state;
         | 
| 993 995 | 
             
            }
         | 
    
        data/ext/bootsnap/extconf.rb
    CHANGED
    
    | @@ -1,21 +1,23 @@ | |
| 1 1 | 
             
            # frozen_string_literal: true
         | 
| 2 | 
            +
             | 
| 2 3 | 
             
            require("mkmf")
         | 
| 3 4 |  | 
| 4 | 
            -
            if RUBY_ENGINE ==  | 
| 5 | 
            -
              $CFLAGS <<  | 
| 6 | 
            -
              $CFLAGS <<  | 
| 5 | 
            +
            if RUBY_ENGINE == "ruby"
         | 
| 6 | 
            +
              $CFLAGS << " -O3 "
         | 
| 7 | 
            +
              $CFLAGS << " -std=c99"
         | 
| 7 8 |  | 
| 8 9 | 
             
              # ruby.h has some -Wpedantic fails in some cases
         | 
| 9 10 | 
             
              # (e.g. https://github.com/Shopify/bootsnap/issues/15)
         | 
| 10 | 
            -
              unless [ | 
| 11 | 
            -
                $CFLAGS <<  | 
| 12 | 
            -
                $CFLAGS <<  | 
| 13 | 
            -
                $CFLAGS <<  | 
| 14 | 
            -
                $CFLAGS <<  | 
| 11 | 
            +
              unless ["0", "", nil].include?(ENV["BOOTSNAP_PEDANTIC"])
         | 
| 12 | 
            +
                $CFLAGS << " -Wall"
         | 
| 13 | 
            +
                $CFLAGS << " -Werror"
         | 
| 14 | 
            +
                $CFLAGS << " -Wextra"
         | 
| 15 | 
            +
                $CFLAGS << " -Wpedantic"
         | 
| 15 16 |  | 
| 16 | 
            -
                $CFLAGS <<  | 
| 17 | 
            -
                $CFLAGS <<  | 
| 18 | 
            -
                $CFLAGS <<  | 
| 17 | 
            +
                $CFLAGS << " -Wno-unused-parameter" # VALUE self has to be there but we don't care what it is.
         | 
| 18 | 
            +
                $CFLAGS << " -Wno-keyword-macro" # hiding return
         | 
| 19 | 
            +
                $CFLAGS << " -Wno-gcc-compat" # ruby.h 2.6.0 on macos 10.14, dunno
         | 
| 20 | 
            +
                $CFLAGS << " -Wno-compound-token-split-by-macro"
         | 
| 19 21 | 
             
              end
         | 
| 20 22 |  | 
| 21 23 | 
             
              create_makefile("bootsnap/bootsnap")
         | 
    
        data/lib/bootsnap/bundler.rb
    CHANGED
    
    
    
        data/lib/bootsnap/cli.rb
    CHANGED
    
    | @@ -1,10 +1,10 @@ | |
| 1 1 | 
             
            # frozen_string_literal: true
         | 
| 2 2 |  | 
| 3 | 
            -
            require  | 
| 4 | 
            -
            require  | 
| 5 | 
            -
            require  | 
| 6 | 
            -
            require  | 
| 7 | 
            -
            require  | 
| 3 | 
            +
            require "bootsnap"
         | 
| 4 | 
            +
            require "bootsnap/cli/worker_pool"
         | 
| 5 | 
            +
            require "optparse"
         | 
| 6 | 
            +
            require "fileutils"
         | 
| 7 | 
            +
            require "etc"
         | 
| 8 8 |  | 
| 9 9 | 
             
            module Bootsnap
         | 
| 10 10 | 
             
              class CLI
         | 
| @@ -21,51 +21,58 @@ module Bootsnap | |
| 21 21 |  | 
| 22 22 | 
             
                attr_reader :cache_dir, :argv
         | 
| 23 23 |  | 
| 24 | 
            -
                attr_accessor :compile_gemfile, :exclude, :verbose, :iseq, :yaml, :jobs
         | 
| 24 | 
            +
                attr_accessor :compile_gemfile, :exclude, :verbose, :iseq, :yaml, :json, :jobs
         | 
| 25 25 |  | 
| 26 26 | 
             
                def initialize(argv)
         | 
| 27 27 | 
             
                  @argv = argv
         | 
| 28 | 
            -
                  self.cache_dir = ENV.fetch( | 
| 28 | 
            +
                  self.cache_dir = ENV.fetch("BOOTSNAP_CACHE_DIR", "tmp/cache")
         | 
| 29 29 | 
             
                  self.compile_gemfile = false
         | 
| 30 30 | 
             
                  self.exclude = nil
         | 
| 31 31 | 
             
                  self.verbose = false
         | 
| 32 32 | 
             
                  self.jobs = Etc.nprocessors
         | 
| 33 33 | 
             
                  self.iseq = true
         | 
| 34 34 | 
             
                  self.yaml = true
         | 
| 35 | 
            +
                  self.json = true
         | 
| 35 36 | 
             
                end
         | 
| 36 37 |  | 
| 37 38 | 
             
                def precompile_command(*sources)
         | 
| 38 | 
            -
                  require  | 
| 39 | 
            -
                  require  | 
| 39 | 
            +
                  require "bootsnap/compile_cache/iseq"
         | 
| 40 | 
            +
                  require "bootsnap/compile_cache/yaml"
         | 
| 41 | 
            +
                  require "bootsnap/compile_cache/json"
         | 
| 40 42 |  | 
| 41 43 | 
             
                  fix_default_encoding do
         | 
| 42 | 
            -
                    Bootsnap::CompileCache::ISeq.cache_dir =  | 
| 44 | 
            +
                    Bootsnap::CompileCache::ISeq.cache_dir = cache_dir
         | 
| 43 45 | 
             
                    Bootsnap::CompileCache::YAML.init!
         | 
| 44 | 
            -
                    Bootsnap::CompileCache::YAML.cache_dir =  | 
| 46 | 
            +
                    Bootsnap::CompileCache::YAML.cache_dir = cache_dir
         | 
| 47 | 
            +
                    Bootsnap::CompileCache::JSON.init!
         | 
| 48 | 
            +
                    Bootsnap::CompileCache::JSON.cache_dir = cache_dir
         | 
| 45 49 |  | 
| 46 50 | 
             
                    @work_pool = WorkerPool.create(size: jobs, jobs: {
         | 
| 47 51 | 
             
                      ruby: method(:precompile_ruby),
         | 
| 48 52 | 
             
                      yaml: method(:precompile_yaml),
         | 
| 53 | 
            +
                      json: method(:precompile_json),
         | 
| 49 54 | 
             
                    })
         | 
| 50 55 | 
             
                    @work_pool.spawn
         | 
| 51 56 |  | 
| 52 57 | 
             
                    main_sources = sources.map { |d| File.expand_path(d) }
         | 
| 53 58 | 
             
                    precompile_ruby_files(main_sources)
         | 
| 54 59 | 
             
                    precompile_yaml_files(main_sources)
         | 
| 60 | 
            +
                    precompile_json_files(main_sources)
         | 
| 55 61 |  | 
| 56 62 | 
             
                    if compile_gemfile
         | 
| 57 63 | 
             
                      # Some gems embed their tests, they're very unlikely to be loaded, so not worth precompiling.
         | 
| 58 | 
            -
                      gem_exclude = Regexp.union([exclude,  | 
| 64 | 
            +
                      gem_exclude = Regexp.union([exclude, "/spec/", "/test/"].compact)
         | 
| 59 65 | 
             
                      precompile_ruby_files($LOAD_PATH.map { |d| File.expand_path(d) }, exclude: gem_exclude)
         | 
| 60 66 |  | 
| 61 | 
            -
                      # Gems that include YAML files usually don't put them in `lib/`.
         | 
| 67 | 
            +
                      # Gems that include JSON or YAML files usually don't put them in `lib/`.
         | 
| 62 68 | 
             
                      # So we look at the gem root.
         | 
| 63 69 | 
             
                      gem_pattern = %r{^#{Regexp.escape(Bundler.bundle_path.to_s)}/?(?:bundler/)?gems\/[^/]+}
         | 
| 64 70 | 
             
                      gem_paths = $LOAD_PATH.map { |p| p[gem_pattern] }.compact.uniq
         | 
| 65 71 | 
             
                      precompile_yaml_files(gem_paths, exclude: gem_exclude)
         | 
| 72 | 
            +
                      precompile_json_files(gem_paths, exclude: gem_exclude)
         | 
| 66 73 | 
             
                    end
         | 
| 67 74 |  | 
| 68 | 
            -
                    if exitstatus = @work_pool.shutdown
         | 
| 75 | 
            +
                    if (exitstatus = @work_pool.shutdown)
         | 
| 69 76 | 
             
                      exit(exitstatus)
         | 
| 70 77 | 
             
                    end
         | 
| 71 78 | 
             
                  end
         | 
| @@ -82,7 +89,7 @@ module Bootsnap | |
| 82 89 | 
             
                if dir_sort
         | 
| 83 90 | 
             
                  def list_files(path, pattern)
         | 
| 84 91 | 
             
                    if File.directory?(path)
         | 
| 85 | 
            -
                      Dir[File.join(path, | 
| 92 | 
            +
                      Dir[File.join(path, pattern), sort: false]
         | 
| 86 93 | 
             
                    elsif File.exist?(path)
         | 
| 87 94 | 
             
                      [path]
         | 
| 88 95 | 
             
                    else
         | 
| @@ -92,7 +99,7 @@ module Bootsnap | |
| 92 99 | 
             
                else
         | 
| 93 100 | 
             
                  def list_files(path, pattern)
         | 
| 94 101 | 
             
                    if File.directory?(path)
         | 
| 95 | 
            -
                      Dir[File.join(path, | 
| 102 | 
            +
                      Dir[File.join(path, pattern)]
         | 
| 96 103 | 
             
                    elsif File.exist?(path)
         | 
| 97 104 | 
             
                      [path]
         | 
| 98 105 | 
             
                    else
         | 
| @@ -119,9 +126,9 @@ module Bootsnap | |
| 119 126 |  | 
| 120 127 | 
             
                  load_paths.each do |path|
         | 
| 121 128 | 
             
                    if !exclude || !exclude.match?(path)
         | 
| 122 | 
            -
                      list_files(path,  | 
| 129 | 
            +
                      list_files(path, "**/*.{yml,yaml}").each do |yaml_file|
         | 
| 123 130 | 
             
                        # We ignore hidden files to not match the various .ci.yml files
         | 
| 124 | 
            -
                        if !File.basename(yaml_file).start_with?( | 
| 131 | 
            +
                        if !File.basename(yaml_file).start_with?(".") && (!exclude || !exclude.match?(yaml_file))
         | 
| 125 132 | 
             
                          @work_pool.push(:yaml, yaml_file)
         | 
| 126 133 | 
             
                        end
         | 
| 127 134 | 
             
                      end
         | 
| @@ -131,18 +138,41 @@ module Bootsnap | |
| 131 138 |  | 
| 132 139 | 
             
                def precompile_yaml(*yaml_files)
         | 
| 133 140 | 
             
                  Array(yaml_files).each do |yaml_file|
         | 
| 134 | 
            -
                    if CompileCache::YAML.precompile(yaml_file | 
| 141 | 
            +
                    if CompileCache::YAML.precompile(yaml_file)
         | 
| 135 142 | 
             
                      STDERR.puts(yaml_file) if verbose
         | 
| 136 143 | 
             
                    end
         | 
| 137 144 | 
             
                  end
         | 
| 138 145 | 
             
                end
         | 
| 139 146 |  | 
| 147 | 
            +
                def precompile_json_files(load_paths, exclude: self.exclude)
         | 
| 148 | 
            +
                  return unless json
         | 
| 149 | 
            +
             | 
| 150 | 
            +
                  load_paths.each do |path|
         | 
| 151 | 
            +
                    if !exclude || !exclude.match?(path)
         | 
| 152 | 
            +
                      list_files(path, "**/*.json").each do |json_file|
         | 
| 153 | 
            +
                        # We ignore hidden files to not match the various .config.json files
         | 
| 154 | 
            +
                        if !File.basename(json_file).start_with?(".") && (!exclude || !exclude.match?(json_file))
         | 
| 155 | 
            +
                          @work_pool.push(:json, json_file)
         | 
| 156 | 
            +
                        end
         | 
| 157 | 
            +
                      end
         | 
| 158 | 
            +
                    end
         | 
| 159 | 
            +
                  end
         | 
| 160 | 
            +
                end
         | 
| 161 | 
            +
             | 
| 162 | 
            +
                def precompile_json(*json_files)
         | 
| 163 | 
            +
                  Array(json_files).each do |json_file|
         | 
| 164 | 
            +
                    if CompileCache::JSON.precompile(json_file)
         | 
| 165 | 
            +
                      STDERR.puts(json_file) if verbose
         | 
| 166 | 
            +
                    end
         | 
| 167 | 
            +
                  end
         | 
| 168 | 
            +
                end
         | 
| 169 | 
            +
             | 
| 140 170 | 
             
                def precompile_ruby_files(load_paths, exclude: self.exclude)
         | 
| 141 171 | 
             
                  return unless iseq
         | 
| 142 172 |  | 
| 143 173 | 
             
                  load_paths.each do |path|
         | 
| 144 174 | 
             
                    if !exclude || !exclude.match?(path)
         | 
| 145 | 
            -
                      list_files(path,  | 
| 175 | 
            +
                      list_files(path, "**/*.rb").each do |ruby_file|
         | 
| 146 176 | 
             
                        if !exclude || !exclude.match?(ruby_file)
         | 
| 147 177 | 
             
                          @work_pool.push(:ruby, ruby_file)
         | 
| 148 178 | 
             
                        end
         | 
| @@ -153,7 +183,7 @@ module Bootsnap | |
| 153 183 |  | 
| 154 184 | 
             
                def precompile_ruby(*ruby_files)
         | 
| 155 185 | 
             
                  Array(ruby_files).each do |ruby_file|
         | 
| 156 | 
            -
                    if CompileCache::ISeq.precompile(ruby_file | 
| 186 | 
            +
                    if CompileCache::ISeq.precompile(ruby_file)
         | 
| 157 187 | 
             
                      STDERR.puts(ruby_file) if verbose
         | 
| 158 188 | 
             
                    end
         | 
| 159 189 | 
             
                  end
         | 
| @@ -180,7 +210,7 @@ module Bootsnap | |
| 180 210 | 
             
                end
         | 
| 181 211 |  | 
| 182 212 | 
             
                def cache_dir=(dir)
         | 
| 183 | 
            -
                  @cache_dir = File.expand_path(File.join(dir,  | 
| 213 | 
            +
                  @cache_dir = File.expand_path(File.join(dir, "bootsnap/compile-cache"))
         | 
| 184 214 | 
             
                end
         | 
| 185 215 |  | 
| 186 216 | 
             
                def exclude_pattern(pattern)
         | 
| @@ -195,24 +225,24 @@ module Bootsnap | |
| 195 225 | 
             
                    opts.separator "GLOBAL OPTIONS"
         | 
| 196 226 | 
             
                    opts.separator ""
         | 
| 197 227 |  | 
| 198 | 
            -
                    help = <<~ | 
| 228 | 
            +
                    help = <<~HELP
         | 
| 199 229 | 
             
                      Path to the bootsnap cache directory. Defaults to tmp/cache
         | 
| 200 | 
            -
                     | 
| 201 | 
            -
                    opts.on( | 
| 230 | 
            +
                    HELP
         | 
| 231 | 
            +
                    opts.on("--cache-dir DIR", help.strip) do |dir|
         | 
| 202 232 | 
             
                      self.cache_dir = dir
         | 
| 203 233 | 
             
                    end
         | 
| 204 234 |  | 
| 205 | 
            -
                    help = <<~ | 
| 235 | 
            +
                    help = <<~HELP
         | 
| 206 236 | 
             
                      Print precompiled paths.
         | 
| 207 | 
            -
                     | 
| 208 | 
            -
                    opts.on( | 
| 237 | 
            +
                    HELP
         | 
| 238 | 
            +
                    opts.on("--verbose", "-v", help.strip) do
         | 
| 209 239 | 
             
                      self.verbose = true
         | 
| 210 240 | 
             
                    end
         | 
| 211 241 |  | 
| 212 | 
            -
                    help = <<~ | 
| 242 | 
            +
                    help = <<~HELP
         | 
| 213 243 | 
             
                      Number of workers to use. Default to number of processors, set to 0 to disable multi-processing.
         | 
| 214 | 
            -
                     | 
| 215 | 
            -
                    opts.on( | 
| 244 | 
            +
                    HELP
         | 
| 245 | 
            +
                    opts.on("--jobs JOBS", "-j", help.strip) do |jobs|
         | 
| 216 246 | 
             
                      self.jobs = Integer(jobs)
         | 
| 217 247 | 
             
                    end
         | 
| 218 248 |  | 
| @@ -221,25 +251,30 @@ module Bootsnap | |
| 221 251 | 
             
                    opts.separator ""
         | 
| 222 252 | 
             
                    opts.separator "    precompile [DIRECTORIES...]: Precompile all .rb files in the passed directories"
         | 
| 223 253 |  | 
| 224 | 
            -
                    help = <<~ | 
| 254 | 
            +
                    help = <<~HELP
         | 
| 225 255 | 
             
                      Precompile the gems in Gemfile
         | 
| 226 | 
            -
                     | 
| 227 | 
            -
                    opts.on( | 
| 256 | 
            +
                    HELP
         | 
| 257 | 
            +
                    opts.on("--gemfile", help) { self.compile_gemfile = true }
         | 
| 228 258 |  | 
| 229 | 
            -
                    help = <<~ | 
| 259 | 
            +
                    help = <<~HELP
         | 
| 230 260 | 
             
                      Path pattern to not precompile. e.g. --exclude 'aws-sdk|google-api'
         | 
| 231 | 
            -
                     | 
| 232 | 
            -
                    opts.on( | 
| 261 | 
            +
                    HELP
         | 
| 262 | 
            +
                    opts.on("--exclude PATTERN", help) { |pattern| exclude_pattern(pattern) }
         | 
| 233 263 |  | 
| 234 | 
            -
                    help = <<~ | 
| 264 | 
            +
                    help = <<~HELP
         | 
| 235 265 | 
             
                      Disable ISeq (.rb) precompilation.
         | 
| 236 | 
            -
                     | 
| 237 | 
            -
                    opts.on( | 
| 266 | 
            +
                    HELP
         | 
| 267 | 
            +
                    opts.on("--no-iseq", help) { self.iseq = false }
         | 
| 238 268 |  | 
| 239 | 
            -
                    help = <<~ | 
| 269 | 
            +
                    help = <<~HELP
         | 
| 240 270 | 
             
                      Disable YAML precompilation.
         | 
| 241 | 
            -
                     | 
| 242 | 
            -
                    opts.on( | 
| 271 | 
            +
                    HELP
         | 
| 272 | 
            +
                    opts.on("--no-yaml", help) { self.yaml = false }
         | 
| 273 | 
            +
             | 
| 274 | 
            +
                    help = <<~HELP
         | 
| 275 | 
            +
                      Disable JSON precompilation.
         | 
| 276 | 
            +
                    HELP
         | 
| 277 | 
            +
                    opts.on("--no-json", help) { self.json = false }
         | 
| 243 278 | 
             
                  end
         | 
| 244 279 | 
             
                end
         | 
| 245 280 | 
             
              end
         |