capistrano-s3_archive 0.5.4 → 0.9.9

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: d9e6383d1737e4e90cdb9bff196905fe7bcf6b66
-  data.tar.gz: f74ce9246e6fa2e51c1ebdbb4cfe2d6b5982e338
+  metadata.gz: 97cf12d50fcd21483d0594609a57b4867f898e47
+  data.tar.gz: 7109636be3cb0841c520546ca318991c5a6bbf28
 SHA512:
-  metadata.gz: 8d40317da88ca00be017c73fc0ec930dc9f50187af85d3df432a93da6e4604a674d0221e252084b47fba3ee464b9b2b5989822d03ea487ef0e8e64c98496fc4e
-  data.tar.gz: 606f2213cf280af3ce02fa9ea031a15aed91f6707591e103b1bffb00d9f2106dc2bd4899513ad429de33ae1eae7378f4aeeb362b58837eeb0b37b8dca19c7e0c
+  metadata.gz: f0953e0106a31337751b0e0c9eb045ce845581d313ad040a96ab529308a2911c4fc127b182be47ff9d3da80d94a614974163521862e40d2fb0acefef23b24903
+  data.tar.gz: 243cc2aa9fd85e1977a08705827ae26f3a2d10be857bdd1430c6cc47406cbf39c09775c28baff8c5bc298fd37650364d72b786efe5e9b331df24f1e6fe5bfad2

data/README.md

@@ -1,16 +1,19 @@
 # Capistrano::S3Archive
 
-Capistrano::S3Archive is an extention of [Capistrano](http://www.capistranorb.com/) which enables to `set :scm, :s3_archive`.
+Capistrano::S3Archive is an extention of [Capistrano](http://www.capistranorb.com/).
 
 This behaves like the [capistrano-rsync](https://github.com/moll/capistrano-rsync) except downloading sources from S3 instead of GIT by default.
 
+![img_s3archive_to_local_to_remote](./img/s3_archive-rsync.png)
+
+**CAUTION!!** Document for VERSION < 0.9 is [legacy_README](legacy_README.md)
 
 ## Installation
 
 Add this line to your application's Gemfile:
 
 ```ruby
-gem 'capistrano-s3_archive'
+gem 'capistrano-s3_archive', '>= 0.9'
 ```
 
 And then execute:
@@ -23,9 +26,16 @@ And then execute:
 
 ## Usage
 
-`set :scm, :s3_archive` in your config file.
+### Quick Start
+
+In Capfile:
+
+```
+    require "capistrano/scm/s3_archive"
+    install_plugin Capistrano::SCM::S3Archive
+```
 
-Set a S3 path containing source archives to `repo_url`. For example, if you has following tree,
+And set a S3 path containing source archives to `:repo_url` and the parameters to access Amazon S3 to `:s3_archive_client_options`, For example, if you has following tree,
 
     s3://yourbucket/somedirectory/
                       |- 201506011200.zip
@@ -34,15 +44,15 @@ Set a S3 path containing source archives to `repo_url`. For example, if you has
                       |- 201506020100.zip
                       `- 201506030100.zip
 
-then `set :repo_url, 's3://yourbucket/somedirectory'`.
+then your `config/deploy.rb` would be:
 
-Set parameters to access Amazon S3:
-
-```ruby
-set :s3_client_options, { region: 'ap-northeast-1', credentials: somecredentials }
 ```
+    set :repo_url, 's3://yourbucket/somedirectory/'
+    set :s3_archive_client_options, { region: 'ap-northeast-1', credentials: somecredentials }
+```
+
+To deploy staging:
 
-And set regular capistrano options. To deploy staging:
 ```
 $ bundle exec cap staging deploy
 ```
@@ -54,26 +64,69 @@ $ bundle exec cap staging deploy_only
 
 
 ### Configuration
-Set parameters with `set :key, value`.
-
-#### Rsync Strategy (default)
-
-Key           | Default | Description
---------------|---------|------------
-branch        | `latest` | The S3 Object basename to download. Support `:latest` or such as `'201506011500.zip'`.
-version_id    | nil      | Version ID of version-controlled S3 object. It should use with `:branch`. e.g. `set :branch, 'myapp.zip'; set :version_id, 'qawsedrftgyhujikolq'`
-sort_proc     | `->(a,b) { b.key <=> a.key }` | Sort algorithm used to detect `:latest` object basename. It should be proc object for `a,b` as `Aws::S3::Object` comparing.
-rsync_options | `['-az']` | Options used to rsync.
-local_cache   | `tmp/deploy` | Path where to extruct your archive on local for staging and rsyncing. Can be both relative or absolute.
-rsync_cache   | `shared/deploy` | Path where to cache your repository on the server to avoid rsyncing from scratch each time. Can be both relative or absolute.<br> Set to `nil` if you want to disable the cache.
-s3_archive    | `tmp/archives` | Path where to download source archives. Can be both relative or absolute.
-archive_release_runner_options | { in: :groups, limit: fetch(:archive_release_runner_concurrency) } | Runner options on creating releases.
-(archive_release_runner_concurrency) | 20 | Default value of runner concurrency option.
-
-##### Experimental configration
-Key           | Default | Description
---------------|---------|------------
-hardlink      | nil     | Enable `--link-dest` option when remote rsyncing. It could speed deployment up in the case rsync_cache enabled.
+
+Available configurations are followings (key, default).
+
+    :repo_url, nil
+    :branch, :latest
+    :s3_archive_client_options, nil
+    :s3_archive_sort_proc, ->(new, old) { old.key <=> new.key }
+    :s3_archive_object_version_id, nil
+    :s3_archive_local_download_dir, "tmp/archives"
+    :s3_archive_local_cache_dir, "tmp/deploy"
+    :s3_archive_remote_rsync_options, ['-az', '--delete']
+    :s3_archive_remote_rsync_ssh_options, []
+    :s3_archive_remote_rsync_runner_options, {}
+    :s3_archive_rsync_cache_dir, "shared/deploy"
+    :s3_archive_hardlink_release, false
+
+**`repo_url` (required)**
+
+The S3 bucket and prefix where the archives are stored. e.g. 's3://yourbucket/somedirectory/'.
+
+**`branch`**
+
+Basename of archive object to deploy. In the previous example at Quick Start section, you can use `'201506011500.zip'`, `'201506020100.zip'`, etc. And `:latest` is a special symbol to select latest object automatically by `:s3_archive_sort_proc`.
+
+**`s3_archive_client_options` (required)**
+
+Options passed to `Aws::S3::Client.new(options)` to fetch archives.
+
+**`s3_archive_sort_proc`**
+
+Sort algorithm used to detect basename of `:latest` object. It should be proc object for `new,old` as `Aws::S3::Object` comparing.
+
+**`:s3_archive_object_version_id`**
+
+Version ID of version-controlled S3 object. It should use with `:branch`. e.g. `set :branch, 'myapp.zip'; set :version_id, 'qawsedrftgyhujikolq'`
+
+**`:s3_archive_local_download_dir`**
+
+Path where to download source archives. Can use both relative or absolute.
+
+**`:s3_archive_local_cache_dir`**
+
+Path where to extruct your archive on local for staging and rsyncing. Can use both relative or absolute.
+
+**`:s3_archive_remote_rsync_options`**
+
+Options used to rsync to remote cache dir.
+
+**`:s3_archive_remote_rsync_ssh_options`**
+
+Options used in `rsync -e 'ssh OPTIONS'`.
+
+**`:s3_archive_remote_rsync_runner_options`**
+
+Runner options of a task to rsync to remote cache, this options are passed to `on release_roles(:all), options` in the rsyncing task. It's useful when to reduce the overload of the machine running Capistrano. e.g. `set :s3_archive_remote_rsync_runner_options, { in: :groups, limit: 10 }`.
+
+**`:s3_archive_rsync_cache_dir`**
+
+Path where to cache your sources on the remote server to avoid rsyncing from scratch each time. Can use both relative or absolute from `deploy_to` path.
+
+**`:s3_archive_hardlink_release`**
+
+Enable `--link-dest` option when creating release directory by remote rsyncing. It could speed deployment up.
 
 ## Development
 

data/capistrano-s3_archive.gemspec

@@ -19,9 +19,10 @@ Gem::Specification.new do |spec|
   spec.require_paths = ["lib"]
 
   spec.required_ruby_version = '>= 2.0.0'
-  spec.add_dependency 'capistrano', '~> 3.6.0'
-  spec.add_dependency 'aws-sdk-core', '~> 2.0'
+  spec.add_dependency 'capistrano', '~> 3.0'
+  spec.add_dependency 'aws-sdk', '~> 2.0'
 
-  spec.add_development_dependency "bundler", "~> 1.9"
-  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_development_dependency "bundler"
+  spec.add_development_dependency "rake"
+  spec.add_development_dependency "rubocop"
 end

data/legacy_README.md

@@ -0,0 +1,97 @@
+# Capistrano::S3Archive (README for legacy version)
+
+**CAUTION**
+Capistrano plugin system has been renewed at version 3.7. This legacy_README is the document for old version.
+
+https://github.com/capistrano/capistrano/blob/master/UPGRADING-3.7.md
+
+================
+
+Capistrano::S3Archive is an extention of [Capistrano](http://www.capistranorb.com/) which enables to `set :scm, :s3_archive`.
+
+This behaves like the [capistrano-rsync](https://github.com/moll/capistrano-rsync) except downloading sources from S3 instead of GIT by default.
+
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'capistrano-s3_archive'
+```
+
+And then execute:
+
+    $ bundle
+
+<!-- Or install it yourself as: -->
+
+<!--     $ gem install capistrano-s3_archive -->
+
+## Usage
+
+`set :scm, :s3_archive` in your config file.
+
+Set a S3 path containing source archives to `repo_url`. For example, if you has following tree,
+
+    s3://yourbucket/somedirectory/
+                      |- 201506011200.zip
+                      |- 201506011500.zip
+                      ...
+                      |- 201506020100.zip
+                      `- 201506030100.zip
+
+then `set :repo_url, 's3://yourbucket/somedirectory'`.
+
+Set parameters to access Amazon S3:
+
+```ruby
+set :s3_client_options, { region: 'ap-northeast-1', credentials: somecredentials }
+```
+
+And set regular capistrano options. To deploy staging:
+```
+$ bundle exec cap staging deploy
+```
+
+Or to skip download & extruct archive and deploy local files:
+```
+$ bundle exec cap staging deploy_only
+```
+
+
+### Configuration
+Set parameters with `set :key, value`.
+
+#### Rsync Strategy (default)
+
+Key           | Default | Description
+--------------|---------|------------
+branch        | `latest` | The S3 Object basename to download. Support `:latest` or such as `'201506011500.zip'`.
+version_id    | nil      | Version ID of version-controlled S3 object. It should use with `:branch`. e.g. `set :branch, 'myapp.zip'; set :version_id, 'qawsedrftgyhujikolq'`
+sort_proc     | `->(a,b) { b.key <=> a.key }` | Sort algorithm used to detect `:latest` object basename. It should be proc object for `a,b` as `Aws::S3::Object` comparing.
+rsync_options | `['-az']` | Options used to rsync.
+local_cache   | `tmp/deploy` | Path where to extruct your archive on local for staging and rsyncing. Can be both relative or absolute.
+rsync_cache   | `shared/deploy` | Path where to cache your repository on the server to avoid rsyncing from scratch each time. Can be both relative or absolute.<br> Set to `nil` if you want to disable the cache.
+s3_archive    | `tmp/archives` | Path where to download source archives. Can be both relative or absolute.
+archive_release_runner_options | { in: :groups, limit: fetch(:archive_release_runner_concurrency) } | Runner options on creating releases.
+(archive_release_runner_concurrency) | 20 | Default value of runner concurrency option.
+
+##### Experimental configration
+Key           | Default | Description
+--------------|---------|------------
+hardlink      | nil     | Enable `--link-dest` option when remote rsyncing. It could speed deployment up in the case rsync_cache enabled.
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release` to create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+1. Fork it ( https://github.com/[my-github-username]/capistrano-s3_archive/fork )
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create a new Pull Request

data/lib/capistrano/legacy_s3_archive.rb

@@ -0,0 +1,249 @@
+$stderr.puts "DEPRECATION WARNING: `set :scm, :s3_archive` is deprecated. see https://github.com/capistrano/capistrano/blob/master/UPGRADING-3.7.md to update to the plugin architecture."
+
+require 'aws-sdk-core'
+
+load File.expand_path("../tasks/legacy_s3_archive.rake", __FILE__)
+
+require "capistrano/s3_archive/version"
+require 'capistrano/scm'
+
+set_if_empty :rsync_options, ['-az --delete']
+set_if_empty :rsync_ssh_options, []
+set_if_empty :rsync_copy, "rsync --archive --acls --xattrs"
+set_if_empty :rsync_cache, "shared/deploy"
+set_if_empty :local_cache, "tmp/deploy"
+set_if_empty :s3_archive, "tmp/archives"
+set_if_empty :sort_proc, ->(a,b) { b.key <=> a.key }
+set_if_empty :archive_release_runner_concurrency, 20
+set_if_empty :archive_release_runner_options, { in: :groups, limit: fetch(:archive_release_runner_concurrency) }
+
+module Capistrano
+  module S3Archive
+    class SCM < Capistrano::SCM
+      include FileUtils
+      attr_accessor :bucket, :object_prefix
+
+      def initialize(*args)
+        super
+        @bucket, @object_prefix = parse_s3_uri(repo_url)
+        set :local_cache_dir, "#{fetch(:local_cache)}/#{fetch(:stage)}"
+      end
+
+      def get_object(target)
+        opts = { bucket: bucket, key: archive_object_key }
+        opts[:version_id] = fetch(:version_id) if fetch(:version_id)
+        s3_client.get_object(opts, target: target)
+      end
+
+      def get_object_metadata
+        s3_client.list_object_versions(bucket: bucket, prefix: archive_object_key).versions.find do |v|
+          if fetch(:version_id) then v.version_id == fetch(:version_id)
+          else v.is_latest
+          end
+        end
+      end
+
+      def list_objects(all_page = true)
+        response = s3_client.list_objects(bucket: bucket, prefix: object_prefix)
+        if all_page
+          response.inject([]) do |objects, page|
+            objects += page.contents
+          end
+        else
+          response
+        end
+      end
+
+      def archive_object_key
+        @archive_object_key ||=
+          case fetch(:branch).to_sym
+          when :master, :latest, nil
+            latest_object_key
+          else
+            object_prefix + fetch(:branch)
+          end
+      end
+
+      private
+      def s3_client
+        @s3_client ||= Aws::S3::Client.new(fetch(:s3_client_options))
+      end
+
+      def latest_object_key
+        list_objects.sort(&fetch(:sort_proc)).first.key
+      end
+
+      def parse_s3_uri(uri)
+        pathes = uri.split('://')[1].split('/')
+        [pathes.first, pathes.drop(1).push('').join('/')]
+      end
+
+      ### Default strategy
+      private
+      module RsyncStrategy
+        class MissingSSHKyesError < StandardError; end
+        class ResourceBusyError < StandardError; end
+
+        def local_check
+          list_objects(false)
+        end
+
+        def check
+          return if context.class == SSHKit::Backend::Local
+          ssh_key  = ssh_key_for(context.host)
+          if ssh_key.nil?
+            fail MissingSSHKyesError, "#{RsyncStrategy} only supports publickey authentication. Please set #{context.host.hostname}.keys or ssh_options."
+          end
+        end
+
+        def stage
+          stage_lock do
+            archive_dir = File.join(fetch(:s3_archive), fetch(:stage).to_s)
+            archive_file = File.join(archive_dir, File.basename(archive_object_key))
+            tmp_file = "#{archive_file}.part"
+            etag_file = File.join(archive_dir, ".#{File.basename(archive_object_key)}.etag")
+            fail "#{tmp_file} is found. Another process is running?" if File.exist?(tmp_file)
+
+            etag = get_object_metadata.tap { |it| fail "No such object: #{current_revision}" if it.nil? }.etag
+            if [archive_file, etag_file].all? { |f| File.exist?(f) } && File.read(etag_file) == etag
+              context.info "#{archive_file} (etag:#{etag}) is found. download skipped."
+            else
+              context.info "Download #{current_revision} to #{archive_file}"
+              mkdir_p(File.dirname(archive_file))
+              File.open(tmp_file, 'w') do |f|
+                get_object(f)
+              end
+              move(tmp_file, archive_file)
+              File.write(etag_file, etag)
+            end
+
+            remove_entry_secure(fetch(:local_cache_dir)) if File.exist? fetch(:local_cache_dir)
+            mkdir_p(fetch(:local_cache_dir))
+            case archive_file
+            when /\.zip\Z/
+              cmd = "unzip -q -d #{fetch(:local_cache_dir)} #{archive_file}"
+            when /\.tar\.gz\Z|\.tar\.bz2\Z|\.tgz\Z/
+              cmd = "tar xf #{archive_file} -C #{fetch(:local_cache_dir)}"
+            end
+
+            release_lock(true) do
+              run_locally do
+                execute cmd
+              end
+            end
+          end
+        end
+
+        def cleanup
+          run_locally do
+            archives_dir = File.join(fetch(:s3_archive), fetch(:stage).to_s)
+            archives = capture(:ls, '-xtr', archives_dir).split
+            if archives.count >= fetch(:keep_releases)
+              tobe_removes = (archives - archives.last(fetch(:keep_releases)))
+              if tobe_removes.any?
+                tobe_removes_str = tobe_removes.map do |file|
+                  File.join(archives_dir, file)
+                end.join(' ')
+                execute :rm, tobe_removes_str
+              end
+            end
+          end
+        end
+
+        def release(server = context.host)
+          unless context.class == SSHKit::Backend::Local
+            user = user_for(server) + '@' unless user_for(server).nil?
+            key  = ssh_key_for(server)
+            ssh_port_option = server.port.nil? ? '' : "-p #{server.port}"
+          end
+          rsync = ['rsync']
+          rsync.concat fetch(:rsync_options)
+          rsync << fetch(:local_cache_dir) + '/'
+          unless context.class == SSHKit::Backend::Local
+            rsync << "-e 'ssh -i #{key} #{ssh_port_option} #{fetch(:rsync_ssh_options).join(' ')}'"
+            rsync << "#{user}#{server.hostname}:#{rsync_cache || release_path}"
+          else
+            rsync << '--no-compress'
+            rsync << "#{rsync_cache || release_path}"
+          end
+          release_lock do
+            run_locally do
+              execute *rsync
+            end
+          end
+
+          unless fetch(:rsync_cache).nil?
+            cache = rsync_cache
+            link_option = if fetch(:hardlink) && test!("[ `readlink #{current_path}` != #{release_path} ]")
+                            "--link-dest `readlink #{current_path}`"
+                          end
+            copy = %(#{fetch(:rsync_copy)} #{link_option} "#{cache}/" "#{release_path}/")
+            context.execute copy
+          end
+        end
+
+        def current_revision
+          fetch(:version_id) ? "#{archive_object_key}?versionid=#{fetch(:version_id)}" : archive_object_key
+        end
+
+        def ssh_key_for(host)
+          if not host.keys.empty?
+            host.keys.first
+          elsif host.ssh_options && host.ssh_options.has_key?(:keys)
+            Array(host.ssh_options[:keys]).first
+          elsif fetch(:ssh_options, nil) && fetch(:ssh_options).has_key?(:keys)
+            fetch(:ssh_options)[:keys].first
+          end
+        end
+
+        def user_for(host)
+          if host.user
+            host.user
+          elsif host.ssh_options && host.ssh_options.has_key?(:user)
+            host.ssh_options[:user]
+          elsif fetch(:ssh_options, nil) && fetch(:ssh_options).has_key?(:user)
+            fetch(:ssh_options)[:user]
+          end
+        end
+
+        private
+        def rsync_cache
+          cache = fetch(:rsync_cache)
+          cache = deploy_to + "/" + cache if cache && cache !~ /^\//
+          cache
+        end
+
+        def stage_lock(&block)
+          mkdir_p(File.dirname(fetch(:local_cache)))
+          lockfile = "#{fetch(:local_cache)}.#{fetch(:stage)}.lock"
+          File.open(lockfile, "w") do |f|
+            if f.flock(File::LOCK_EX | File::LOCK_NB)
+              block.call
+            else
+              fail ResourceBusyError, "Could not get #{lockfile}"
+            end
+          end
+        ensure
+          rm lockfile if File.exist? lockfile
+        end
+
+        def release_lock(exclusive = false, &block)
+          mkdir_p(File.dirname(fetch(:local_cache)))
+          lockfile = "#{fetch(:local_cache)}.#{fetch(:stage)}.release.lock"
+          File.open(lockfile, File::RDONLY|File::CREAT) do |f|
+            mode = if exclusive
+                     File::LOCK_EX | File::LOCK_NB
+                   else
+                     File::LOCK_SH
+                   end
+            if f.flock(mode)
+              block.call
+            else
+              fail ResourceBusyError, "Could not get #{fetch(:lockfile)}"
+            end
+          end
+        end
+      end
+    end
+  end
+end

data/lib/capistrano/s3_archive.rb

@@ -1,247 +1 @@
-require 'aws-sdk-core'
-
-load File.expand_path("../tasks/s3_archive.rake", __FILE__)
-
-require "capistrano/s3_archive/version"
-require 'capistrano/scm'
-
-set_if_empty :rsync_options, ['-az --delete']
-set_if_empty :rsync_ssh_options, []
-set_if_empty :rsync_copy, "rsync --archive --acls --xattrs"
-set_if_empty :rsync_cache, "shared/deploy"
-set_if_empty :local_cache, "tmp/deploy"
-set_if_empty :s3_archive, "tmp/archives"
-set_if_empty :sort_proc, ->(a,b) { b.key <=> a.key }
-set_if_empty :archive_release_runner_concurrency, 20
-set_if_empty :archive_release_runner_options, { in: :groups, limit: fetch(:archive_release_runner_concurrency) }
-
-module Capistrano
-  module S3Archive
-    class SCM < Capistrano::SCM
-      include FileUtils
-      attr_accessor :bucket, :object_prefix
-
-      def initialize(*args)
-        super
-        @bucket, @object_prefix = parse_s3_uri(repo_url)
-        set :local_cache_dir, "#{fetch(:local_cache)}/#{fetch(:stage)}"
-      end
-
-      def get_object(target)
-        opts = { bucket: bucket, key: archive_object_key }
-        opts[:version_id] = fetch(:version_id) if fetch(:version_id)
-        s3_client.get_object(opts, target: target)
-      end
-
-      def get_object_metadata
-        s3_client.list_object_versions(bucket: bucket, prefix: archive_object_key).versions.find do |v|
-          if fetch(:version_id) then v.version_id == fetch(:version_id)
-          else v.is_latest
-          end
-        end
-      end
-
-      def list_objects(all_page = true)
-        response = s3_client.list_objects(bucket: bucket, prefix: object_prefix)
-        if all_page
-          response.inject([]) do |objects, page|
-            objects += page.contents
-          end
-        else
-          response
-        end
-      end
-
-      def archive_object_key
-        @archive_object_key ||=
-          case fetch(:branch).to_sym
-          when :master, :latest, nil
-            latest_object_key
-          else
-            object_prefix + fetch(:branch)
-          end
-      end
-
-      private
-      def s3_client
-        @s3_client ||= Aws::S3::Client.new(fetch(:s3_client_options))
-      end
-
-      def latest_object_key
-        list_objects.sort(&fetch(:sort_proc)).first.key
-      end
-
-      def parse_s3_uri(uri)
-        pathes = uri.split('://')[1].split('/')
-        [pathes.first, pathes.drop(1).push('').join('/')]
-      end
-
-      ### Default strategy
-      private
-      module RsyncStrategy
-        class MissingSSHKyesError < StandardError; end
-        class ResourceBusyError < StandardError; end
-
-        def local_check
-          list_objects(false)
-        end
-
-        def check
-          return if context.class == SSHKit::Backend::Local
-          ssh_key  = ssh_key_for(context.host)
-          if ssh_key.nil?
-            fail MissingSSHKyesError, "#{RsyncStrategy} only supports publickey authentication. Please set #{context.host.hostname}.keys or ssh_options."
-          end
-        end
-
-        def stage
-          stage_lock do
-            archive_dir = File.join(fetch(:s3_archive), fetch(:stage).to_s)
-            archive_file = File.join(archive_dir, File.basename(archive_object_key))
-            tmp_file = "#{archive_file}.part"
-            etag_file = File.join(archive_dir, ".#{File.basename(archive_object_key)}.etag")
-            fail "#{tmp_file} is found. Another process is running?" if File.exist?(tmp_file)
-
-            etag = get_object_metadata.tap { |it| fail "No such object: #{current_revision}" if it.nil? }.etag
-            if [archive_file, etag_file].all? { |f| File.exist?(f) } && File.read(etag_file) == etag
-              context.info "#{archive_file} (etag:#{etag}) is found. download skipped."
-            else
-              context.info "Download #{current_revision} to #{archive_file}"
-              mkdir_p(File.dirname(archive_file))
-              File.open(tmp_file, 'w') do |f|
-                get_object(f)
-              end
-              move(tmp_file, archive_file)
-              File.write(etag_file, etag)
-            end
-
-            remove_entry_secure(fetch(:local_cache_dir)) if File.exist? fetch(:local_cache_dir)
-            mkdir_p(fetch(:local_cache_dir))
-            case archive_file
-            when /\.zip\Z/
-              cmd = "unzip -q -d #{fetch(:local_cache_dir)} #{archive_file}"
-            when /\.tar\.gz\Z|\.tar\.bz2\Z|\.tgz\Z/
-              cmd = "tar xf #{archive_file} -C #{fetch(:local_cache_dir)}"
-            end
-
-            release_lock(true) do
-              run_locally do
-                execute cmd
-              end
-            end
-          end
-        end
-
-        def cleanup
-          run_locally do
-            archives_dir = File.join(fetch(:s3_archive), fetch(:stage).to_s)
-            archives = capture(:ls, '-xtr', archives_dir).split
-            if archives.count >= fetch(:keep_releases)
-              tobe_removes = (archives - archives.last(fetch(:keep_releases)))
-              if tobe_removes.any?
-                tobe_removes_str = tobe_removes.map do |file|
-                  File.join(archives_dir, file)
-                end.join(' ')
-                execute :rm, tobe_removes_str
-              end
-            end
-          end
-        end
-
-        def release(server = context.host)
-          unless context.class == SSHKit::Backend::Local
-            user = user_for(server) + '@' unless user_for(server).nil?
-            key  = ssh_key_for(server)
-            ssh_port_option = server.port.nil? ? '' : "-p #{server.port}"
-          end
-          rsync = ['rsync']
-          rsync.concat fetch(:rsync_options)
-          rsync << fetch(:local_cache_dir) + '/'
-          unless context.class == SSHKit::Backend::Local
-            rsync << "-e 'ssh -i #{key} #{ssh_port_option} #{fetch(:rsync_ssh_options).join(' ')}'"
-            rsync << "#{user}#{server.hostname}:#{rsync_cache || release_path}"
-          else
-            rsync << '--no-compress'
-            rsync << "#{rsync_cache || release_path}"
-          end
-          release_lock do
-            run_locally do
-              execute *rsync
-            end
-          end
-
-          unless fetch(:rsync_cache).nil?
-            cache = rsync_cache
-            link_option = if fetch(:hardlink) && test!("[ `readlink #{current_path}` != #{release_path} ]")
-                            "--link-dest `readlink #{current_path}`"
-                          end
-            copy = %(#{fetch(:rsync_copy)} #{link_option} "#{cache}/" "#{release_path}/")
-            context.execute copy
-          end
-        end
-
-        def current_revision
-          fetch(:version_id) ? "#{archive_object_key}?versionid=#{fetch(:version_id)}" : archive_object_key
-        end
-
-        def ssh_key_for(host)
-          if not host.keys.empty?
-            host.keys.first
-          elsif host.ssh_options && host.ssh_options.has_key?(:keys)
-            Array(host.ssh_options[:keys]).first
-          elsif fetch(:ssh_options, nil) && fetch(:ssh_options).has_key?(:keys)
-            fetch(:ssh_options)[:keys].first
-          end
-        end
-
-        def user_for(host)
-          if host.user
-            host.user
-          elsif host.ssh_options && host.ssh_options.has_key?(:user)
-            host.ssh_options[:user]
-          elsif fetch(:ssh_options, nil) && fetch(:ssh_options).has_key?(:user)
-            fetch(:ssh_options)[:user]
-          end
-        end
-
-        private
-        def rsync_cache
-          cache = fetch(:rsync_cache)
-          cache = deploy_to + "/" + cache if cache && cache !~ /^\//
-          cache
-        end
-
-        def stage_lock(&block)
-          mkdir_p(File.dirname(fetch(:local_cache)))
-          lockfile = "#{fetch(:local_cache)}.#{fetch(:stage)}.lock"
-          File.open(lockfile, "w") do |f|
-            if f.flock(File::LOCK_EX | File::LOCK_NB)
-              block.call
-            else
-              fail ResourceBusyError, "Could not get #{lockfile}"
-            end
-          end
-        ensure
-          rm lockfile if File.exist? lockfile
-        end
-
-        def release_lock(exclusive = false, &block)
-          mkdir_p(File.dirname(fetch(:local_cache)))
-          lockfile = "#{fetch(:local_cache)}.#{fetch(:stage)}.release.lock"
-          File.open(lockfile, File::RDONLY|File::CREAT) do |f|
-            mode = if exclusive
-                     File::LOCK_EX | File::LOCK_NB
-                   else
-                     File::LOCK_SH
-                   end
-            if f.flock(mode)
-              block.call
-            else
-              fail ResourceBusyError, "Could not get #{fetch(:lockfile)}"
-            end
-          end
-        end
-      end
-    end
-  end
-end
+load File.expand_path("../legacy_s3_archive.rb", __FILE__)

data/lib/capistrano/s3_archive/version.rb

@@ -1,5 +1,5 @@
 module Capistrano
   module S3Archive
-    VERSION = "0.5.4"
+    VERSION = "0.9.9"
   end
 end

data/lib/capistrano/scm/s3_archive.rb

@@ -0,0 +1,308 @@
+require "capistrano/scm/plugin"
+require "aws-sdk"
+require "uri"
+
+module Capistrano
+  class SCM
+    class S3Archive < Capistrano::SCM::Plugin
+      attr_reader :extractor
+      include FileUtils
+
+      class ResourceBusyError < StandardError; end
+
+      def set_defaults
+        set_if_empty :s3_archive_client_options, {}
+        set_if_empty :s3_archive_extract_to, :local # :local or :remote
+        set_if_empty(:s3_archive_sort_proc, ->(new, old) { old.key <=> new.key })
+        set_if_empty :s3_archive_object_version_id, nil
+        set_if_empty :s3_archive_local_download_dir, "tmp/archives"
+        set_if_empty :s3_archive_local_cache_dir, "tmp/deploy"
+        set_if_empty :s3_archive_remote_rsync_options, ['-az', '--delete']
+        set_if_empty :s3_archive_remote_rsync_ssh_options, []
+        set_if_empty :s3_archive_remote_rsync_runner_options, {}
+        set_if_empty :s3_archive_rsync_cache_dir, "shared/deploy"
+        set_if_empty :s3_archive_hardlink_release, false
+        # internal use
+        set_if_empty :s3_archive_rsync_copy, "rsync --archive --acls --xattrs"
+      end
+
+      def define_tasks
+        eval_rakefile File.expand_path("../tasks/s3_archive.rake", __FILE__)
+      end
+
+      def register_hooks
+        after "deploy:new_release_path", "s3_archive:create_release"
+        before "deploy:check", "s3_archive:check"
+        before "deploy:set_current_revision", "s3_archive:set_current_revision"
+      end
+
+      def local_check
+        s3_client.list_objects(bucket: s3params.bucket, prefix: s3params.object_prefix)
+      end
+
+      def get_object(target)
+        opts = { bucket: s3params.bucket, key: archive_object_key }
+        opts[:version_id] = fetch(:s3_archive_object_version_id) if fetch(:s3_archive_object_version_id)
+        s3_client.get_object(opts, target: target)
+      end
+
+      def remote_check
+        backend.execute :echo, 'check ssh'
+      end
+
+      def stage
+        stage_lock do
+          archive_dir = File.join(fetch(:s3_archive_local_download_dir), fetch(:stage).to_s)
+          archive_file = File.join(archive_dir, File.basename(archive_object_key))
+          tmp_file = "#{archive_file}.part"
+          etag_file = File.join(archive_dir, ".#{File.basename(archive_object_key)}.etag")
+          fail "#{tmp_file} is found. Another process is running?" if File.exist?(tmp_file)
+          etag = get_object_metadata.tap { |it| fail "No such object: #{current_revision}" if it.nil? }.etag
+
+
+          if [archive_file, etag_file].all? { |f| File.exist?(f) } && File.read(etag_file) == etag
+            backend.info "#{archive_file} (etag:#{etag}) is found. download skipped."
+          else
+            backend.info "Download #{current_revision} to #{archive_file}"
+            mkdir_p(File.dirname(archive_file))
+            File.open(tmp_file, 'w') do |f|
+              get_object(f)
+            end
+            move(tmp_file, archive_file)
+            File.write(etag_file, etag)
+          end
+
+          remove_entry_secure(fetch(:s3_archive_local_cache_dir)) if File.exist? fetch(:s3_archive_local_cache_dir)
+          mkdir_p(fetch(:s3_archive_local_cache_dir))
+          case archive_file
+          when /\.zip\Z/
+            cmd = "unzip -q -d #{fetch(:s3_archive_local_cache_dir)} #{archive_file}"
+          when /\.tar\.gz\Z|\.tar\.bz2\Z|\.tgz\Z/
+            cmd = "tar xf #{archive_file} -C #{fetch(:s3_archive_local_cache_dir)}"
+          end
+
+          release_lock_on_stage do
+            run_locally do
+              execute cmd
+            end
+          end
+        end
+      end
+
+      def cleanup_stage_dir
+        run_locally do
+          archives_dir = File.join(fetch(:s3_archive_local_download_dir), fetch(:stage).to_s)
+          archives = capture(:ls, '-xtr', archives_dir).split
+          if archives.count >= fetch(:keep_releases)
+            to_be_removes = (archives - archives.last(fetch(:keep_releases)))
+            if to_be_removes.any?
+              to_be_removes_str = to_be_removes.map do |file|
+                File.join(archives_dir, file)
+              end.join(' ')
+              execute :rm, to_be_removes_str
+            end
+          end
+        end
+      end
+
+      def transfer_sources(dest)
+        fail "#{__method__} must be called in run_locally" unless backend.is_a?(SSHKit::Backend::Local)
+
+        rsync = ['rsync']
+        rsync.concat fetch(:s3_archive_remote_rsync_options, [])
+        rsync << (fetch(:s3_archive_local_cache_dir) + '/')
+
+        if dest.local?
+          rsync << ('--no-compress')
+          rsync << rsync_cache_dir
+        else
+          rsync << "-e 'ssh #{dest.ssh_key_option} #{fetch(:s3_archive_remote_rsync_ssh_options).join(' ')}'"
+          rsync << "#{dest.login_user_at}#{dest.hostname}:#{rsync_cache_dir}"
+        end
+
+        release_lock_on_create do
+          backend.execute(*rsync)
+        end
+      end
+
+      def release
+        link_option = if fetch(:s3_archive_hardlink_release) && backend.test("[ `readlink #{current_path}` != #{release_path} ]")
+                        "--link-dest `readlink #{current_path}`"
+                      end
+        create_release = %[#{fetch(:s3_archive_rsync_copy)} #{link_option} "#{rsync_cache_dir}/" "#{release_path}/"]
+        backend.execute create_release
+      end
+
+      def current_revision
+        if fetch(:s3_archive_object_version_id)
+          "#{archive_object_key}?versionid=#{fetch(:s3_archive_object_version_id)}"
+        else
+          archive_object_key
+        end
+      end
+
+      def archive_object_key
+        @archive_object_key ||=
+          case fetch(:branch, :latest).to_sym
+          when :master, :latest
+            latest_object_key
+          else
+            s3params.object_prefix + fetch(:branch).to_s
+          end
+      end
+
+      def rsync_cache_dir
+        File.join(deploy_to, fetch(:s3_archive_rsync_cache_dir))
+      end
+
+      def s3params
+        @s3params ||= S3Params.new(fetch(:repo_url))
+      end
+
+      def get_object_metadata
+        s3_client.list_object_versions(bucket: s3params.bucket, prefix: archive_object_key).versions.find do |v|
+          if fetch(:s3_archive_object_version_id) then v.version_id == fetch(:s3_archive_object_version_id)
+          else v.is_latest
+          end
+        end
+      end
+
+      def list_all_objects
+        response = s3_client.list_objects(bucket: s3params.bucket, prefix: s3params.object_prefix)
+        response.inject([]) do |objects, page|
+          objects + page.contents
+        end
+      end
+
+      def latest_object_key
+        list_all_objects.sort(&fetch(:s3_archive_sort_proc)).first.key
+      end
+
+      private
+
+      def release_lock_on_stage(&block)
+        release_lock((File::LOCK_EX | File::LOCK_NB), &block) # exclusive lock
+      end
+
+      def release_lock_on_create(&block)
+        release_lock(File::LOCK_SH, &block)
+      end
+
+      def release_lock(lock_mode, &block)
+        mkdir_p(File.dirname(fetch(:s3_archive_local_cache_dir)))
+        lockfile = "#{fetch(:s3_archive_local_cache_dir)}.#{fetch(:stage)}.release.lock"
+        File.open(lockfile, File::RDONLY | File::CREAT) do |file|
+          if file.flock(lock_mode)
+            block.call
+          else
+            fail ResourceBusyError, "Could not get #{lockfile}"
+          end
+        end
+      end
+
+      def stage_lock(&block)
+        mkdir_p(File.dirname(fetch(:s3_archive_local_cache_dir)))
+        lockfile = "#{fetch(:s3_archive_local_cache_dir)}.#{fetch(:stage)}.lock"
+        File.open(lockfile, "w") do |file|
+          fail ResourceBusyError, "Could not get #{lockfile}" unless file.flock(File::LOCK_EX | File::LOCK_NB)
+          block.call
+        end
+      ensure
+        rm lockfile if File.exist? lockfile
+      end
+
+      def s3_client
+        @s3_client ||= Aws::S3::Client.new(fetch(:s3_archive_client_options))
+      end
+
+       class LocalExtractor
+        # class ResourceBusyError < StandardError; end
+
+        # include FileUtils
+
+        def stage
+          stage_lock do
+            archive_dir = File.join(fetch(:s3_archive_local_download_dir), fetch(:stage).to_s)
+            archive_file = File.join(archive_dir, File.basename(archive_object_key))
+            tmp_file = "#{archive_file}.part"
+            etag_file = File.join(archive_dir, ".#{File.basename(archive_object_key)}.etag")
+            fail "#{tmp_file} is found. Another process is running?" if File.exist?(tmp_file)
+            etag = get_object_metadata.tap { |it| fail "No such object: #{current_revision}" if it.nil? }.etag
+
+
+            if [archive_file, etag_file].all? { |f| File.exist?(f) } && File.read(etag_file) == etag
+              context.info "#{archive_file} (etag:#{etag}) is found. download skipped."
+            else
+              context.info "Download #{current_revision} to #{archive_file}"
+              mkdir_p(File.dirname(archive_file))
+              File.open(tmp_file, 'w') do |f|
+                get_object(f)
+              end
+              move(tmp_file, archive_file)
+              File.write(etag_file, etag)
+            end
+
+            remove_entry_secure(fetch(:s3_archive_local_cache_dir)) if File.exist? fetch(:s3_archive_local_cache_dir)
+            mkdir_p(fetch(:s3_archive_local_cache_dir))
+            case archive_file
+            when /\.zip\Z/
+              cmd = "unzip -q -d #{fetch(:s3_archive_local_cache_dir)} #{archive_file}"
+            when /\.tar\.gz\Z|\.tar\.bz2\Z|\.tgz\Z/
+              cmd = "tar xf #{archive_file} -C #{fetch(:s3_archive_local_cache_dir)}"
+            end
+
+            release_lock_on_stage do
+              run_locally do
+                execute cmd
+              end
+            end
+          end
+        end
+
+        def stage_lock(&block)
+          mkdir_p(File.dirname(fetch(:s3_archive_local_cache_dir)))
+          lockfile = "#{fetch(:s3_archive_local_cache_dir)}.#{fetch(:stage)}.lock"
+          begin
+            File.open(lockfile, "w") do |file|
+              fail ResourceBusyError, "Could not get #{lockfile}" unless file.flock(File::LOCK_EX | File::LOCK_NB)
+              block.call
+            end
+          ensure
+            rm lockfile if File.exist? lockfile
+          end
+        end
+      end
+
+      class RemoteExtractor
+      end
+
+      class S3Params
+        attr_reader :bucket, :object_prefix
+
+        def initialize(repo_url)
+          uri = URI.parse(repo_url)
+          @bucket = uri.host
+          @object_prefix = uri.path.sub(/\/?\Z/, '/').slice(1..-1) # normalize path
+        end
+      end
+    end
+  end
+
+  class Configuration
+    class Server
+      def login_user_at
+        user = [user, ssh_options[:user]].compact.first
+        user ? "#{user}@" : ''
+      end
+
+      def ssh_key_option
+        key = [keys, ssh_options[:keys]].flatten.compact.first
+        key ? "-i #{key}" : ''
+      end
+
+      def ssh_port_option
+        port ? "-p #{port}" : ''
+      end
+    end
+  end
+end

data/lib/capistrano/scm/tasks/s3_archive.rake

@@ -0,0 +1,57 @@
+plugin = self
+
+namespace :s3_archive do
+  desc 'Check that the S3 buckets are reachable'
+  task :check do
+    run_locally do
+      plugin.local_check
+    end
+
+    on release_roles :all do
+      plugin.remote_check
+    end
+  end
+
+  desc 'Extruct and stage the S3 archive in a stage directory'
+  task :stage do
+    if fetch(:skip_staging, false)
+      info "Skip extracting and staging."
+      next
+    end
+
+    run_locally do
+      plugin.stage
+    end
+  end
+
+  after :stage, :cleanup_stage_dir do
+    run_locally do
+      plugin.cleanup_stage_dir
+    end
+  end
+
+  desc 'Copy repo to releases'
+  task create_release: :stage do
+    on release_roles(:all), fetch(:s3_archive_remote_rsync_runner_options) do |server|
+      test "[ -e #{plugin.rsync_cache_dir} ]" # implicit initialize for 'server'
+      run_locally do
+        plugin.transfer_sources(server)
+      end
+    end
+
+    on release_roles(:all) do
+      execute :mkdir, '-p', release_path
+      plugin.release
+    end
+  end
+
+  desc 'Determine the revision that will be deployed'
+  task :set_current_revision do
+    set :current_revision, plugin.current_revision
+  end
+end unless Rake::Task.task_defined?("s3_archive:check")
+
+task :deploy_only do
+  set :skip_staging, true
+  invoke :deploy
+end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: capistrano-s3_archive
 version: !ruby/object:Gem::Version
-  version: 0.5.4
+  version: 0.9.9
 platform: ruby
 authors:
 - Takuto Komazaki
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2017-03-10 00:00:00.000000000 Z
+date: 2017-07-13 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: capistrano
@@ -16,16 +16,16 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 3.6.0
+        version: '3.0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 3.6.0
+        version: '3.0'
 - !ruby/object:Gem::Dependency
-  name: aws-sdk-core
+  name: aws-sdk
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
@@ -42,30 +42,44 @@ dependencies:
   name: bundler
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '1.9'
+        version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '1.9'
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '10.0'
+        version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rubocop
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '10.0'
+        version: '0'
 description: Capistrano deployment from an archive on Amazon S3.
 email:
 - komazarari@gmail.com
@@ -82,9 +96,14 @@ files:
 - bin/console
 - bin/setup
 - capistrano-s3_archive.gemspec
+- img/s3_archive-rsync.png
+- legacy_README.md
+- lib/capistrano/legacy_s3_archive.rb
 - lib/capistrano/s3_archive.rb
 - lib/capistrano/s3_archive/version.rb
-- lib/capistrano/tasks/s3_archive.rake
+- lib/capistrano/scm/s3_archive.rb
+- lib/capistrano/scm/tasks/s3_archive.rake
+- lib/capistrano/tasks/legacy_s3_archive.rake
 - vagrant_example/.insecure_private_key
 - vagrant_example/Capfile
 - vagrant_example/Gemfile
@@ -109,7 +128,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.6.8
+rubygems_version: 2.6.11
 signing_key: 
 specification_version: 4
 summary: Capistrano deployment from an archive on Amazon S3.