prebundler 0.0.2

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 2cb3b9c01592cb314beb744cfe49b05daf72f4e7
+  data.tar.gz: acb37cc9d4fc50a278758ff20cb17746f17c864d
+SHA512:
+  metadata.gz: 5aed2eee4563955bf5bc7f568a1b162b117c24d4ef06e62c38f3167901d42a0d2d5e92762866330535c914b737eb7a51aa499f924c516e9d72c2432c50b5e506
+  data.tar.gz: 9d0a8bd2b1953742cff48b7155f2d5901b80e595fbf33a98ffae8ca52b55e5191c538d6a0d97b6d60cd2ae4418a36c13eed9c6a1028df63ffcb0719787fd4885

data/CHANGELOG.md

@@ -0,0 +1,7 @@
+0.0.2
+===
+- Better CLI interface.
+
+0.0.1
+===
+- Birthday!

data/Gemfile

@@ -0,0 +1,12 @@
+source 'https://rubygems.org'
+
+gemspec
+
+group :development, :test do
+  gem 'pry-byebug'
+  gem 'rake'
+end
+
+group :test do
+  gem 'rspec', '~> 3.0'
+end

data/README.md

@@ -0,0 +1,93 @@
+## prebundler
+Speed up gem installation by prebuilding gems and storing them in S3.
+
+## Installation
+
+`gem install prebundler`
+
+### Why?
+
+If you've ever worked on a large application that has hundreds of dependencies, you have probably felt the pain of running `bundle install` for the first time - it takes forever! This is especially true in the Docker world, where you're basically running `bundle install` for the first time every time. All your dependencies have to be reinstalled even if just one of them changes. This is because Docker images are built as a series of layers. If a layer changes, for example because it contains an altered Gemfile.lock, the entire layer (and all the layers that come after) have to be rebuilt.
+
+Unfortunately bundler doesn't really have a great way of mitigating the problem of having to constantly reinstall a huge number of gems. Sure, you can pass the `--jobs` flag to `bundle install` which will fetch gems in parallel, but because of Ruby's global interpreter lock only the I/O parts of gem installation can be truly parallelized. The slowest gems to install are the ones that need to build native extensions (i.e. extensions written in C) as part of the installation process, which is entirely CPU-bound and can't be parallelized.
+
+So what are your options?
+
+1. **Depend on fewer gems**. Totally valid but often impractical, especially since you may depend on a lot of gems _transiently_, meaning they're pulled in by the dependencies of your dependencies.
+
+2. **Vendor everything**. In other words, store copies of all your gems in the `vendor` directory. That way, you don't have to even run `bundle install` during the Docker build. It'll work great as long as you use the same operating system and architecture in development, production, staging, etc or don't depend on gems with native extensions. By and large, native extensions must be built specifically for the architecture they will run on, meaning the same native extension that runs on Mac OS will not run on Ubuntu Linux.
+
+3. **Bear the pain**. Sure, you could use long `bundle install` times as an [excuse to have fun](https://xkcd.com/303/), but if bundling is, for example, blocking you from releasing a new feature or hotfix then you might be hoping we can do better. Hint: we can.
+
+### Enter Prebundler
+
+Prebundler speeds up gem dependency installation by caching the results of installing each gem, including the results from building native extensions. The first time you run prebundler, it:
+
+1. Installs each gem in Gemfile.lock one at a time
+2. Puts the resulting files into a tarball (eg. nokogiri.tar)
+3. Uploads the tarballs to S3 or some other storage backend
+
+The second time you run prebundler, it:
+
+1. Sucks down gems en masse from S3 (i.e. lots of parallel I/O)
+2. Untars the gems straight onto the filesystem
+3. Runs `bundle check` and falls back to `bundle install` if anything's missing
+
+### Ok, but does it work?
+
+For the main repository behind lumosity.com which contains 445 gem dependencies, prebundler reduced the amount of time spent bundling from 6 minutes 7 seconds to 43 seconds on Travis CI. That's an 88% speed increase. On my laptop with 16 cores and an enterprise-grade internet connection, prebundler finished installing the same set of 445 gems in a little over 15 seconds.
+
+Oh I see. You were asking if it installs the gems correctly. The answer is yes. Running `bundle check` found no missing dependencies, and the app can successfully boot and serve traffic.
+
+Ah sorry. You were asking if it's production ready. I'm not ready to make that claim. Prebundler needs to be tested by more people with more diverse use cases before I might consider it production ready. You can help me with that.
+
+### You've convinced me. How do I give Prebundler a try?
+
+So glad you asked. First, create a file called `.prebundle_config` in your repo, usually in the same directory as your Gemfile.lock:
+
+```ruby
+Prebundler.configure do |config|
+  config.storage_backend = Prebundler::S3Backend.new(
+    access_key_id: ENV['S3_ACCESS_KEY'],
+    secret_access_key: ENV['S3_SECRET_KEY'],
+    bucket: 'my-sweet-bucket',
+    region: 'us-east-1'  # or whatever
+  )
+end
+```
+
+Next, modify your Dockerfile to use prebundler instead of bundler (although bundler should be installed too):
+
+```dockerfile
+ARG S3_ACCESS_KEY=required           # placeholder
+ARG S3_SECRET_KEY=required           # placeholder
+ENV S3_ACCESS_KEY ${S3_ACCESS_KEY}   # copy build arg into ENV
+ENV S3_SECRET_KEY ${S3_SECRET_KEY}   # copy build arg into ENV
+
+RUN gem install bundler
+RUN gem install prebundler && prebundle install
+```
+
+Now when you build your Docker image, you'll need to pass two additional build arguments:
+
+```sh
+docker build \
+  --build-arg S3_ACCESS_KEY=... \
+  --build-arg S3_SECRET_KEY=... \
+  -t registry.tld/organization/repo:tag \
+  ./
+```
+
+Then sit back and watch the magic, keeping in mind of course that the first time you do a Docker build prebundler will be extremely slow. Run it again and watch it fly.
+
+### Disclaimer
+
+Remember, everybody's applications are a little different. What works for one doesn't necessarily work for the other. If you don't depend on that many gems with native extensions, then prebundler might not help you very much. I guess what I'm trying to say is, your mileage may vary.
+
+## License
+
+Licensed under the MIT license. See LICENSE for details.
+
+## Authors
+
+* Cameron C. Dutro: http://github.com/camertron

data/Rakefile

@@ -0,0 +1,18 @@
+# encoding: UTF-8
+
+require 'rubygems' unless ENV['NO_RUBYGEMS']
+
+require 'bundler'
+require 'rspec/core/rake_task'
+require 'rubygems/package_task'
+
+require 'prebundler'
+
+Bundler::GemHelper.install_tasks
+
+task default: :spec
+
+desc 'Run specs'
+RSpec::Core::RakeTask.new do |t|
+  t.pattern = './spec/**/*_spec.rb'
+end

data/bin/prebundle

@@ -0,0 +1,87 @@
+#! /usr/bin/env ruby
+
+require 'gli'
+require 'etc'
+require 'bundler'
+require 'prebundler'
+require 'prebundler/version'
+
+# @TODO: remove
+require 'pry-byebug'
+
+$out = Prebundler::WritePipe.new
+
+include GLI::App
+
+program_desc 'Gem dependency prebuilder'
+
+version Prebundler::VERSION
+
+subcommand_option_handling :normal
+arguments :strict
+
+desc "Don't log to stdout"
+switch [:s, :silent]
+
+desc 'Path to config file.'
+default_value './.prebundle_config'
+flag [:c, :config]
+
+desc 'Install gems from the Gemfile.lock.'
+command :install do |c|
+  c.desc 'Maximum number of parallel gem installs.'
+  c.default_value Etc.nprocessors
+  c.flag [:j, :jobs], type: Integer
+
+  c.desc 'Path to the gemfile to install gems from.'
+  c.default_value ENV.fetch('BUNDLE_GEMFILE', './Gemfile')
+  c.flag [:g, :gemfile]
+
+  c.desc 'Path to the bundle installation directory.'
+  c.default_value ENV.fetch('BUNDLE_PATH', Bundler.bundle_path.to_s)
+  c.flag [:b, :'bundle-path']
+
+  c.action do |global_options, options, args|
+    raise 'Must specify a non-zero number of jobs' if options[:jobs] < 1
+    Prebundler::Cli::Install.run($out, global_options, options, args)
+  end
+end
+
+desc 'List each gem and associated source.'
+command :list do |c|
+  c.desc 'Path to the gemfile.'
+  c.default_value ENV.fetch('BUNDLE_GEMFILE', './Gemfile')
+  c.flag [:g, :gemfile]
+
+  c.desc 'Filter by source. Will perform partial matching.'
+  c.flag [:s, :source], multiple: true
+
+  c.action do |global_options, options, args|
+    Prebundler::Cli::List.run($out, global_options, options, args)
+  end
+end
+
+pre do |global, command, options, args|
+  # Pre logic here
+  # Return true to proceed; false to abort and not call the
+  # chosen command
+  # Use skips_pre before a command to skip this block
+  # on that command only
+  $out.silence! if global[:silent]
+  load global[:config]
+  true
+end
+
+post do |global, command, options, args|
+  # Post logic here
+  # Use skips_post before a command to skip this
+  # block on that command only
+end
+
+on_error do |exception|
+  # Error logic here
+  # return false to skip default error handling
+  true
+end
+
+exit run(ARGV)

data/lib/prebundler/cli/base.rb

@@ -0,0 +1,24 @@
+module Prebundler
+  module Cli
+    class Base
+      class << self
+        def run(out, global_options, options, args)
+          new(out, global_options, options, args).run
+        end
+      end
+
+      attr_reader :out, :global_options, :options, :args
+
+      def initialize(out, global_options, options, args)
+        @out = out
+        @global_options = global_options
+        @options = options
+        @args = args
+      end
+
+      def run
+        raise NotImplementedError, "please define '#{__method__}' in derived classes"
+      end
+    end
+  end
+end

data/lib/prebundler/cli/install.rb

@@ -0,0 +1,105 @@
+require 'securerandom'
+require 'tmpdir'
+require 'parallel'
+require 'bundler'
+
+module Prebundler
+  module Cli
+    class Install < Base
+      def run
+        prepare
+        install
+        check
+      end
+
+      private
+
+      def prepare
+        gem_list.each do |_, gem_ref|
+          next if backend_file_list.include?(gem_ref.tar_file)
+          install_gem(gem_ref) if gem_ref.installable?
+        end
+      end
+
+      def install
+        if options[:jobs] <= 1
+          install_in_serial
+        else
+          install_in_parallel
+        end
+      end
+
+      def install_in_serial
+        gem_list.each do |_, gem_ref|
+          install_gem_ref(gem_ref)
+        end
+      end
+
+      def install_in_parallel
+        Parallel.each(gem_list, in_processes: options[:jobs]) do |_, gem_ref|
+          install_gem_ref(gem_ref)
+        end
+      end
+
+      def install_gem_ref(gem_ref)
+        return unless gem_ref.installable?
+
+        unless File.exist?(gem_ref.install_dir)
+          install_gem(gem_ref)
+        end
+
+        out.puts "Installed #{gem_ref.id}"
+      end
+
+      def install_gem(gem_ref)
+        dest_file = File.join(Dir.tmpdir, "#{SecureRandom.hex}.tar")
+
+        if backend_file_list.include?(gem_ref.tar_file)
+          out.puts "Installing #{gem_ref.id} from backend"
+          config.storage_backend.retrieve_file(gem_ref.tar_file, dest_file)
+          gem_ref.install_from_tar(dest_file)
+          FileUtils.rm(dest_file)
+        else
+          out.puts "Installing #{gem_ref.id} from source"
+          gem_ref.install
+          store_gem(gem_ref, dest_file) if gem_ref.storable?
+        end
+      end
+
+      def store_gem(gem_ref, dest_file)
+        out.puts "Storing #{gem_ref.id}"
+        gem_ref.add_to_tar(dest_file)
+        config.storage_backend.store_file(dest_file, gem_ref.tar_file)
+      end
+
+      def check
+        system 'bundle check'
+
+        if $?.exitstatus != 0
+          out.puts 'Bundle not satisfied, falling back to `bundle install`'
+          system 'bundle install'
+        end
+      end
+
+      def gem_list
+        @gem_list ||= Prebundler::GemfileInterpreter.interpret(gemfile_path, bundle_path)
+      end
+
+      def backend_file_list
+        @backend_file_list ||= config.storage_backend.list_files
+      end
+
+      def gemfile_path
+        options.fetch(:gemfile)
+      end
+
+      def bundle_path
+        options.fetch(:'bundle-path')
+      end
+
+      def config
+        Prebundler.config
+      end
+    end
+  end
+end

data/lib/prebundler/cli/list.rb

@@ -0,0 +1,31 @@
+module Prebundler
+  module Cli
+    class List < Base
+      def run
+        gem_list.each do |_, gem_ref|
+          next unless show_gem?(gem_ref)
+          out.puts "#{gem_ref.id} from #{gem_ref.source}"
+        end
+      end
+
+      private
+
+      def show_gem?(gem_ref)
+        return true if options[:source].empty?
+        options[:source].any? { |source| gem_ref.source.include?(source) }
+      end
+
+      def gem_list
+        @gem_list ||= Prebundler::GemfileInterpreter.interpret(gemfile_path, bundle_path)
+      end
+
+      def gemfile_path
+        options.fetch(:gemfile)
+      end
+
+      def bundle_path
+        nil  # not necessary for resolution
+      end
+    end
+  end
+end

data/lib/prebundler/cli.rb

@@ -0,0 +1,7 @@
+module Prebundler
+  module Cli
+    autoload :Base,    'prebundler/cli/base'
+    autoload :Install, 'prebundler/cli/install'
+    autoload :List,    'prebundler/cli/list'
+  end
+end

data/lib/prebundler/configurator.rb

@@ -0,0 +1,11 @@
+require 'shellwords'
+
+module Prebundler
+  class Configurator
+    attr_accessor :storage_backend, :gem_sources
+
+    def initialize
+      @gem_sources = []
+    end
+  end
+end

data/lib/prebundler/file_backend.rb

@@ -0,0 +1,28 @@
+require 'fileutils'
+
+module Prebundler
+  class FileBackend
+    attr_reader :local_path, :docker_mount_point
+
+    def initialize(options = {})
+      @local_path = options.fetch(:local_path)
+      @docker_mount_point = options.fetch(:docker_mount_point)
+    end
+
+    def store_file(source_file, dest_file)
+      FileUtils.cp(source_file, File.join(docker_mount_point, dest_file))
+    end
+
+    def retrieve_file(source_file, dest_file)
+      FileUtils.cp(File.join(local_path, source_file), dest_file)
+    end
+
+    def list_files
+      Dir.chdir(docker_mount_point) { Dir.glob('**/*.*') }
+    end
+
+    def docker_flags
+      ['-v', "#{File.expand_path(local_path)}:#{docker_mount_point}"]
+    end
+  end
+end

data/lib/prebundler/gem_ref.rb

@@ -0,0 +1,111 @@
+require 'fileutils'
+
+module Prebundler
+  class GemRef
+    REF_TYPES = [PathGemRef, GitGemRef]
+    DEFAULT_SOURCE = 'https://rubygems.org'
+
+    class << self
+      def create(name, bundle_path, options = {})
+        ref_type = REF_TYPES.find { |rt| rt.accepts?(options) } || self
+        ref_type.new(name, bundle_path, options)
+      end
+    end
+
+    attr_reader :name, :bundle_path
+    attr_accessor :spec, :dependencies
+
+    def initialize(name, bundle_path, options = {})
+      @name = name
+      @bundle_path = bundle_path
+      @groups = options[:groups]
+      @source = options[:source]
+      @dependencies = options[:dependencies]
+    end
+
+    def dependencies
+      @dependencies ||= []
+    end
+
+    def groups
+      @groups ||= []
+    end
+
+    def source
+      @source ||= DEFAULT_SOURCE
+    end
+
+    def id
+      "#{name}-#{version}"
+    end
+
+    def version
+      spec.version.to_s
+    end
+
+    def install
+      system "gem install -N --ignore-dependencies --source #{source} #{name} -v #{version}"
+      $?.exitstatus == 0
+    end
+
+    def install_from_tar(tar_file)
+      system "tar -C #{bundle_path} -xf #{tar_file}"
+      $?.exitstatus == 0
+    end
+
+    def add_to_tar(tar_file)
+      tar_flags = File.exist?(tar_file) ? '-rf' : '-cf'
+
+      system "tar -C #{bundle_path} #{tar_flags} #{tar_file} #{relative_gem_dir}"
+      system "tar -C #{bundle_path} -rf #{tar_file} #{relative_gemspec_dir}"
+
+      if File.directory?(extension_dir)
+        system "tar -C #{bundle_path} -rf #{tar_file} #{relative_extension_dir}"
+      end
+    end
+
+    def installable?
+      true
+    end
+
+    def storable?
+      true
+    end
+
+    def install_path
+      File.join(bundle_path, 'gems')
+    end
+
+    def spec_path
+      File.join(bundle_path, 'specifications')
+    end
+
+    def install_dir
+      File.join(install_path, id)
+    end
+
+    def extension_dir
+      File.join(bundle_path, relative_extension_dir)
+    end
+
+    def relative_extension_dir
+      File.join('extensions', Bundler.local_platform.to_s, Gem.extension_api_version.to_s, id)
+    end
+
+    def relative_gem_dir
+      File.join('gems', id)
+    end
+
+    def relative_gemspec_dir
+      File.join('specifications', gemspec_file)
+    end
+
+    def tar_file
+      File.join(Bundler.local_platform.to_s, Gem.extension_api_version.to_s, "#{id}.tar")
+    end
+
+    def gemspec_file
+      "#{id}.gemspec"
+    end
+  end
+end

data/lib/prebundler/gemfile.rb

@@ -0,0 +1,36 @@
+require 'tsort'
+
+module Prebundler
+  class Gemfile
+    include TSort
+    include Enumerable
+
+    attr_reader :gems
+
+    def initialize(gems)
+      @gems = gems
+    end
+
+    def each
+      return to_enum(__method__) unless block_given?
+
+      tsort_each do |name|
+        yield name, gems[name]
+      end
+    end
+
+    alias_method :each_pair, :each
+
+    private
+
+    def tsort_each_node(&block)
+      gems.keys.each(&block)
+    end
+
+    def tsort_each_child(name, &block)
+      gems[name].dependencies.each do |dep|
+        yield dep if gems.include?(dep)
+      end
+    end
+  end
+end

data/lib/prebundler/gemfile_interpreter.rb

@@ -0,0 +1,59 @@
+module Prebundler
+  class GemfileInterpreter
+    def self.interpret(path, bundle_path)
+      Gemfile.new(new(path, bundle_path).gems)
+    end
+
+    attr_reader :gems, :bundle_path
+
+    def initialize(path, bundle_path)
+      @gems = {}
+      @current_groups = []
+      @bundle_path = bundle_path
+      instance_eval(File.read(path))
+
+      lockfile = Bundler::LockfileParser.new(File.read("#{path}.lock"))
+
+      lockfile.specs.each do |spec|
+        gems[spec.name] ||= GemRef.create(spec.name, bundle_path)
+        gems[spec.name].spec = spec
+        gems[spec.name].dependencies = spec.dependencies.map(&:name)
+      end
+    end
+
+    def current_context
+      {
+        path: @current_path,
+        groups: @current_groups,
+        source: @current_source
+      }
+    end
+
+    def gem(name, *args)
+      options = args.find { |a| a.is_a?(Hash) } || {}
+      gems[name] = GemRef.create(name, bundle_path, current_context.merge(options))
+    end
+
+    def path(dir)
+      @current_path = dir
+      yield if block_given?
+      @current_path = nil
+    end
+
+    def source(url)
+      @current_source = url
+      yield if block_given?
+      @current_source = nil
+    end
+
+    def group(*groups)
+      @current_groups = groups
+      yield if block_given?
+      @current_groups = []
+    end
+
+    def gemspec
+      # do nothing
+    end
+  end
+end

data/lib/prebundler/git_gem_ref.rb

@@ -0,0 +1,106 @@
+require 'fileutils'
+require 'uri'
+require 'digest/sha1'
+
+module Prebundler
+  class GitGemRef < GemRef
+    class << self
+      def accepts?(options)
+        options.include?(:git) || options.include?(:github)
+      end
+    end
+
+    attr_reader :strategy
+
+    def initialize(name, bundle_path, options = {})
+      super
+      @strategy = options.include?(:git) ? :git : :github
+      @uri = options[@strategy]
+    end
+
+    def install
+      return if File.exist?(cache_dir) || File.exist?(install_dir)
+      system "git clone #{uri} \"#{cache_dir}\" --bare --no-hardlinks --quiet"
+      return $? if $?.exitstatus != 0
+      system "git clone --no-checkout --quiet \"#{cache_dir}\" \"#{install_dir}\""
+      return $? if $?.exitstatus != 0
+      Dir.chdir(install_dir) { system "git reset --hard --quiet #{revision}" }
+      serialize_gemspecs
+      copy_gemspecs
+      $?
+    end
+
+    def version
+      revision[0...12]
+    end
+
+    def installable?
+      true
+    end
+
+    def storable?
+      false
+    end
+
+    def install_path
+      File.join(bundle_path, 'bundler', 'gems')
+    end
+
+    def cache_path
+      File.join(bundle_path, 'cache', 'bundler', 'git')
+    end
+
+    def cache_dir
+      File.join(cache_path, "#{name}-#{uri_hash}")
+    end
+
+    def uri
+      if strategy == :github
+        "git://github.com/#{@uri}.git"
+      else
+        @uri
+      end
+    end
+
+    alias_method :source, :uri
+
+    def revision
+      spec.source.revision
+    end
+
+    private
+
+    def copy_gemspecs
+      FileUtils.cp(Dir[File.join(install_dir, '*.gemspec')], spec_path)
+    end
+
+    # adapted from
+    # https://github.com/bundler/bundler/blob/fea23637886c1b1bde471c98344b8844f82e60ce/lib/bundler/source/git.rb#L237
+    def serialize_gemspecs
+      Dir[File.join(install_dir, '*.gemspec')].each do |path|
+        # Evaluate gemspecs and cache the result. Gemspecs
+        # in git might require git or other dependencies.
+        # The gemspecs we cache should already be evaluated.
+        spec = Bundler.load_gemspec(path)
+        next unless spec
+        Bundler.rubygems.set_installed_by_version(spec)
+        # Bundler.rubygems.validate(spec)
+        File.open(path, 'wb') { |file| file.write(spec.to_ruby) }
+      end
+    end
+
+    # copied from
+    # https://github.com/bundler/bundler/blob/fea23637886c1b1bde471c98344b8844f82e60ce/lib/bundler/source/git.rb#L281
+    def uri_hash
+      if uri =~ %r{^\w+://(\w+@)?}
+        # Downcase the domain component of the URI
+        # and strip off a trailing slash, if one is present
+        input = URI.parse(uri).normalize.to_s.sub(%r{/$}, "")
+      else
+        # If there is no URI scheme, assume it is an ssh/git URI
+        input = uri
+      end
+      Digest::SHA1.hexdigest(input)
+    end
+  end
+end

data/lib/prebundler/path_gem_ref.rb

@@ -0,0 +1,26 @@
+module Prebundler
+  class PathGemRef < GemRef
+    class << self
+      def accepts?(options)
+        !!options[:path]
+      end
+    end
+
+    attr_reader :path
+
+    def initialize(name, bundle_path, options = {})
+      super
+      @path = options[:path]
+    end
+
+    def installable?
+      false
+    end
+
+    def storable?
+      false
+    end
+
+    alias_method :source, :path
+  end
+end

data/lib/prebundler/s3_backend.rb

@@ -0,0 +1,70 @@
+require 'aws-sdk'
+
+module Prebundler
+  class S3Backend
+    attr_reader :access_key_id, :secret_access_key, :bucket, :region
+
+    def initialize(options = {})
+      @access_key_id = options.fetch(:access_key_id)
+      @secret_access_key = options.fetch(:secret_access_key)
+      @bucket = options.fetch(:bucket)
+      @region = options.fetch(:region)
+    end
+
+    def store_file(source_file, dest_file)
+      File.open(source_file) do |io|
+        client.put_object(bucket: bucket, key: dest_file, body: io)
+      end
+    end
+
+    def retrieve_file(source_file, dest_file)
+      client.get_object(
+        bucket: bucket,
+        key: source_file,
+        response_target: dest_file
+      )
+    end
+
+    def list_files
+      truncated = true
+      continuation_token = nil
+      files = []
+      base_options = {
+        bucket: bucket,
+        prefix: "#{Bundler.local_platform.to_s}/#{Gem.extension_api_version.to_s}"
+      }
+
+      while truncated
+        options = if continuation_token
+          base_options.merge(continuation_token: continuation_token)
+        else
+          base_options
+        end
+
+        response = client.list_objects_v2(options)
+        truncated = response.is_truncated
+        continuation_token = response.continuation_token
+
+        response.contents.each do |file|
+          files << file.key
+        end
+      end
+
+      files
+    end
+
+    def docker_flags
+      []
+    end
+
+    private
+
+    def client
+      @client ||= Aws::S3::Client.new(region: region, credentials: credentials)
+    end
+
+    def credentials
+      @credentials ||= Aws::Credentials.new(access_key_id, secret_access_key)
+    end
+  end
+end

data/lib/prebundler/version.rb

@@ -0,0 +1,3 @@
+module Prebundler
+  VERSION = '0.0.2'
+end

data/lib/prebundler/write_pipe.rb

@@ -0,0 +1,26 @@
+module Prebundler
+  class WritePipe
+    def initialize
+      @silent = false
+    end
+
+    def write(text)
+      return if silent?
+      STDOUT.write(text)
+    end
+
+    def puts(text)
+      return if silent?
+      STDOUT.write("#{text}\n")
+    end
+
+    def silence!
+      @silent = true
+      self
+    end
+
+    def silent?
+      !!@silent
+    end
+  end
+end

data/lib/prebundler.rb

@@ -0,0 +1,26 @@
+module Prebundler
+  autoload :Cli,                'prebundler/cli'
+  autoload :Configurator,       'prebundler/configurator'
+  autoload :FileBackend,        'prebundler/file_backend'
+  autoload :PathGemRef,         'prebundler/path_gem_ref'
+  autoload :Gemfile,            'prebundler/gemfile'
+  autoload :GemfileInterpreter, 'prebundler/gemfile_interpreter'
+  autoload :GemRef,             'prebundler/gem_ref'
+  autoload :GitGemRef,          'prebundler/git_gem_ref'
+  autoload :S3Backend,          'prebundler/s3_backend'
+  autoload :WritePipe,          'prebundler/write_pipe'
+
+  class << self
+    attr_reader :config
+
+    def configure
+      return if configured?
+      @config = Configurator.new
+      yield @config
+    end
+
+    def configured?
+      !!@config
+    end
+  end
+end

data/prebundler.gemspec

@@ -0,0 +1,28 @@
+$:.unshift File.join(File.dirname(__FILE__), 'lib')
+require 'prebundler/version'
+
+Gem::Specification.new do |s|
+  s.name     = 'prebundler'
+  s.version  = ::Prebundler::VERSION
+  s.authors  = ['Cameron Dutro']
+  s.email    = ['camertron@gmail.com']
+  s.homepage = 'http://github.com/camertron'
+
+  s.description = s.summary = 'Gem dependency prebuilder'
+
+  s.platform = Gem::Platform::RUBY
+  s.has_rdoc = true
+
+  s.add_dependency 'bundler'
+  s.add_dependency 'parallel', '~> 1.0'
+  s.add_dependency 'gli', '~> 2.0'
+
+  # @TODO: remove these, maybe move s3 support into separate gem?
+  s.add_dependency 'aws-sdk', '~> 2.0'
+  s.add_dependency 'pry-byebug'
+
+  s.executables << 'prebundle'
+
+  s.require_path = 'lib'
+  s.files = Dir['{lib,spec}/**/*', 'Gemfile', 'CHANGELOG.md', 'README.md', 'Rakefile', 'prebundler.gemspec']
+end

metadata

@@ -0,0 +1,135 @@
+--- !ruby/object:Gem::Specification
+name: prebundler
+version: !ruby/object:Gem::Version
+  version: 0.0.2
+platform: ruby
+authors:
+- Cameron Dutro
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2017-11-29 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: parallel
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+- !ruby/object:Gem::Dependency
+  name: gli
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+- !ruby/object:Gem::Dependency
+  name: aws-sdk
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+- !ruby/object:Gem::Dependency
+  name: pry-byebug
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: Gem dependency prebuilder
+email:
+- camertron@gmail.com
+executables:
+- prebundle
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- Gemfile
+- README.md
+- Rakefile
+- bin/prebundle
+- lib/prebundler.rb
+- lib/prebundler/cli.rb
+- lib/prebundler/cli/base.rb
+- lib/prebundler/cli/install.rb
+- lib/prebundler/cli/list.rb
+- lib/prebundler/configurator.rb
+- lib/prebundler/file_backend.rb
+- lib/prebundler/gem_ref.rb
+- lib/prebundler/gemfile.rb
+- lib/prebundler/gemfile_interpreter.rb
+- lib/prebundler/git_gem_ref.rb
+- lib/prebundler/path_gem_ref.rb
+- lib/prebundler/s3_backend.rb
+- lib/prebundler/version.rb
+- lib/prebundler/write_pipe.rb
+- prebundler.gemspec
+homepage: http://github.com/camertron
+licenses: []
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.6.13
+signing_key: 
+specification_version: 4
+summary: Gem dependency prebuilder
+test_files: []