mamiya 0.2.1 → 0.2.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 5b090d33b53e168ee13379616652d080bc4bd461
-  data.tar.gz: 18514e6676da70a2e6fb15cca785e5c974202027
+  metadata.gz: 3fff2664bac4156374129f9094fe5fc7459d3b8b
+  data.tar.gz: efe0195812cbb482507dae05cacbb05cf8ca57a1
 SHA512:
-  metadata.gz: 93457344b41f4febc0e835d9ad70bb8158602579d13edd078d248da359e898bea7464e54bb4abc7b619d4c7c6935de39a93f9cbc4d2772f5833e1daf930ce2c8
-  data.tar.gz: 99dc4da9eb96dae58a2526fdb36b61aa77dac64a929bb8337f7a1263b6c6f7f2ab048172d1aa9161220975fd88f84bb477f731edd5f32b6679333a5ecc568995
+  metadata.gz: ed7ec72684c57cd099a95e2c66bca126637dd8837eb5c66ea2b9b74fc6a289b87acf041359289319ff79684ed3e6e3ebd56266d3facf1d14b21dd80cebb86083
+  data.tar.gz: aa23567e2219203fb190e5436398c5ae934baceb0ced7dcb7e4e1c226cfeb83c404c32fbb617d358fa37e9e31657556352e8f89caaf9a4384e02ed9aec1206b1

data/README.md

@@ -2,23 +2,17 @@
 
 [![Build Status](https://travis-ci.org/sorah/mamiya.png?branch=master)](https://travis-ci.org/sorah/mamiya)
 
-Mamiya allows you to deploy using without ssh -- using tarballs, some storages (S3, etc), and [Serf](http://www.serfdom.io/).
 
-## Installation
+## What's mamiya?
 
-(Detailed documentation coming soon)
+Mamiya allows you to deploy without ssh -- using tarballs, some storages (S3, etc), and [Serf](http://www.serfdom.io/).
+Build application package on CI or somewhere, then distribute earlier (before you say "Deploy!"). You can switch to a new application quickly, when you're saying "deploy."
 
-1. Install gem (`gem install mamiya`, or bundling by `gem 'mamiya'`)
-2. Prepare master node for `mamiya master` process
-  - master has HTTP API to allow control the cluster via HTTP
-  - master watches agents in the cluster
-3. Then install `mamiya agent` in your deployment target hosts
-4. Write deploy script for your own
-  - how to build package, how to prepare release, how to release, etc.
-5. Build then push package
-  - `mamiya build`
-  - `mamiya push path/to/package.tar.gz`
-6. Time to deploy: `mamiya client deploy -a app package`
+Mamiya uses similar directory structure with Capistrano 'deploy_to' -- you can try this easy.
+
+## Quick Start
+
+See [./docs/quick_start.md](./docs/quick_start.md).
 
 ### Example configuration
 
@@ -30,7 +24,7 @@ Try Mamiya in your local machine: `foreman start`
 Existing major deploy tools (capistrano, mina, ...) use SSH to operate servers.
 But connecting to lot of servers via SSH makes deployment slow.
 
-This solves such problem by using [Serf](http://www.serfdom.io/) and tarball on one storage (Amazon S3).
+Mamiya solves such problem by using [Serf](http://www.serfdom.io/) and tarball on one storage (Amazon S3).
 
 Also, I'm planning to allow to distribute files before the deploy command. I guess this can skip or shorten
 file transferring phase in deploy.

data/docs/quick_start.md

@@ -0,0 +1,212 @@
+# Quick Start
+
+## What's mamiya?
+
+Mamiya allows you to deploy using without ssh -- using tarballs, some storages (S3, etc), and [Serf](http://www.serfdom.io/).
+Build application package on CI or somewhere, then distribute earlier (before you say "Deploy!"). You can switch to an new application quickly, when you're saying "deploy."
+
+Also, this free you from SSH and RSync problems.
+
+## Concepts
+
+- __Package__ is a tarball contains your application.
+- __Storage__ is a place to store package.
+- __Script__ is a Ruby script that defines how to deploy (prepare, restart, etc) your application.
+
+- __CI (or, somewhere)__ builds packages and push them on _Storage_
+- __Master__ controls the cluster of _Agents_
+- __Agent__ receives deployment and do it.
+
+## Prepare master and agent
+
+Run `sudo gem install mamiya` to install mamiya on your systems.
+
+(TODO: write init.d/systemd/upstart script for easier boot up)
+
+### Master
+
+On a server for master node; it'll controll the cluster:
+
+#### Configure
+
+typical configuration for the master here:
+
+``` ruby
+# /etc/mamiya/config.rb
+
+set :storage, {
+  type: :s3,
+  bucket: 'YOUR-S3-BUCKET-NAME',
+  region: 'ap-northeast-1', # set it to your region
+  access_key_id: '...',
+  secret_access_key: '...',
+}
+
+set :serf, {
+  agent: {
+    bind: "0.0.0.0:7760",
+    rpc_addr: "127.0.0.1:7762",
+  }
+}
+
+set :web, {
+  port: 7761,
+}
+```
+
+#### Run it
+
+```
+mamiya master -c /etc/mamiya/config.rb
+```
+
+### Agent
+
+On a server for agent node; where the package will be deployed:
+
+#### Configure
+
+typical configuration for the agents here:
+
+``` ruby
+# /etc/mamiya/config.rb
+
+set :storage, {
+  type: :s3,
+  bucket: 'YOUR-S3-BUCKET-NAME',
+  region: 'ap-northeast-1', # set it to your region
+  access_key_id: '...',
+  secret_access_key: '...',
+}
+
+set :serf, {
+  agent: {
+    bind: "0.0.0.0:7760",
+    rpc_addr: "127.0.0.1:7762",
+    join: 'YOUR-MASTER-HOST:7760', # set it to your master host
+  }
+}
+
+set :web, {
+  port: 7761,
+}
+
+# Where should the application go. :myapp is a Symbol to identify application.
+applications[:myapp] = {deploy_to: '/home/app/myapp'}
+
+# Where should Mamiya store packages, pre-releases temporarily.
+set :packages_dir, '/tmp/mamiya/packages'
+set :prereleases_dir, '/tmp/mamiya/prereleases'
+
+# And how many you want to keep them?
+set :keep_packages, 3
+set :keep_prereleases, 3
+
+# Label your agent. Block will be called every time when referencing labels.
+# (Recommend to implement thread or timeout for periodically fetch...)
+labels do
+  %i(app production active)
+end
+```
+
+#### Run it
+
+```
+mamiya master -c /etc/mamiya/config.rb
+```
+
+### Confirm both working
+
+Use `mamiya client` command family to communicate with master process. By default `http://localhost:7761/` is used. You can change by `-m`, `--master` option or `$MAMIYA_MASTER_URL` environment variable.
+
+```
+master $ mamiya client list-agents
+AGENT_NAME	alive
+```
+
+## Prepare deploy script
+
+_Script_ should have required operations and configurations for an application being deployed.
+
+Script is used for package building and deployments. It'll be copied into a package during package build, then deliver, used on agents.
+
+```
+set :application, 'myapp'
+
+# Using for package build
+set :exclude_from_package, ['tmp', 'log', 'spec', '.sass-cache']
+set :dereference_symlinks, true
+set :build_from, "/tmp/build_from"
+set :build_to, "./builds"
+# set :package_under, 'dir'
+
+# You may use helpers here
+#  set :repository, '...'
+#  use :git, exclude_git_clean_targets: true
+
+# Procedure for build
+build 'write built at' do
+  # build something...
+  File.write('built_at', "#{Time.now}\n")
+end
+
+# Step `prepare` on agents,
+prepare 'write prepared_at' do
+  File.write release_path.join('prepared_at'), Time.now
+end
+
+# You can declare multiples. Run by order defined.
+prepare 'bundle stuff' do
+  run 'bundle', 'install'
+end
+
+# Step `release` run when Release is required. Usually for restarting app process, etc.
+# Also these step declaration accepts labels for `only` and `except` key to limit agents to run on.
+# (Labels can be set by agent's configuration)
+release 'reload unicorn', only [:app] do
+  run 'pkill', '-HUP', '-f', 'unicorn'
+end
+```
+
+## Build your first package and push it
+
+Have all agents and master node run? Okay, let's build your first package!
+
+```
+$ mamiya build -S ./deploy.rb
+11/18 09:37:01   INFO  [Build] Running script.before_build
+11/18 09:37:01   INFO  [Build] Running script.prepare_build
+11/18 09:37:01   INFO  [Build] Running script.build
+11/18 09:37:01   INFO  [Build] Copying script files
+11/18 09:37:01   INFO  [Build] Packed.
+11/18 09:37:01   INFO  [Build] Running script.after_build
+11/18 09:37:01   INFO  [Build] DONE: 20141118093701-myapp built at .../builds/20141118093701-myapp.tar.gz
+```
+
+(`-S`,` --script` specifies where the script is. More options available; see `mamiya help build`)
+
+Great! You can unpack using `tar xf` to confirm what has packed, if you want.
+
+You have to push it onto storage to make available for agents.
+
+```
+$ mamiya push -C ./config.rb
+11/18 09:42:43   INFO  [Push] Pushing builds/20141118093701-myapp.tar.gz to storage(app=myapp)...
+11/18 09:42:43   INFO  [Push] DONE!
+```
+
+(`-C`, `--config` specifies where configuration file is... to find `:storage` configuration.)
+
+## Deploy it
+
+`mamiya client deploy <PACKAGE>` prepares the package then release it after all nodes get ready.
+
+```
+$ export MAMIYA_APP=myapp
+$ mamiya client list-packages
+20141118093701-myapp
+$ mamiya client deploy 20141118093701-myapp
+```
+
+(You can use `-a`, `--application` option instead of `$MAMIYA_APP`)
+

data/lib/mamiya/cli/client.rb

@@ -295,19 +295,17 @@ not distributed: #{dist['not_distributed_count']} agents
 
         if type == :deploy
           do_prep = -> do
-            puts "=> Preparing..."
+            puts " * sending prepare request"
             prepare(package)
           end
 
-          do_prep[]
-
-          puts " * Wait until prepared"
+          puts "=> Wait agents to have prepared"
           puts ""
 
           i = 0
           loop do
             i += 1
-            do_prep[] if i % 25 == 0
+            do_prep[] if i == 2 || i % 25 == 0
 
             s = pkg_status(package, :short)
             puts ""

data/lib/mamiya/storages/s3.rb

@@ -6,6 +6,18 @@ require 'json'
 module Mamiya
   module Storages
     class S3 < Mamiya::Storages::Abstract
+      class MultipleObjectsDeletionError < StandardError
+        attr_accessor :errors
+
+        def initialize(errors)
+          message = errors.map do |error|
+            "#{error.code}: #{error.message} (key=#{error.key})"
+          end.join(', ')
+          super(message)
+          @errors = errors
+        end
+      end
+
       def self.find(config={})
         s3 = initiate_s3_with_config(config)
         Hash[s3.list_objects(bucket: config[:bucket], delimiter: '/').common_prefixes.map { |prefix|
@@ -96,7 +108,10 @@ module Mamiya
         }.compact
         raise NotFound if objs_to_delete.empty?
 
-        s3.delete_objects(bucket: @config[:bucket], delete: {objects: objs_to_delete})
+        result = s3.delete_objects(bucket: @config[:bucket], delete: {objects: objs_to_delete})
+        unless result.errors.empty?
+          raise MultipleObjectsDeletionError.new(result.errors)
+        end
       end
 
       def self.initiate_s3_with_config(config) # :nodoc:

data/lib/mamiya/version.rb

@@ -1,3 +1,3 @@
 module Mamiya
-  VERSION = "0.2.1"
+  VERSION = "0.2.2"
 end

data/spec/storages/s3_spec.rb

@@ -250,6 +250,7 @@ describe Mamiya::Storages::S3 do
 
   describe "#remove(package_name)" do
     let(:package_name) { 'test' }
+    let(:s3_delete_response) { double('response for POST /?delete', errors: []) }
     subject(:remove) { storage.remove(package_name) }
 
     before do
@@ -257,7 +258,7 @@ describe Mamiya::Storages::S3 do
     end
 
     it "removes specified package from S3" do
-      expect(s3).to receive(:delete_objects).with(bucket: 'testbucket', delete: {objects: [{key: 'myapp/test.tar.gz'}, {key: 'myapp/test.json'}]})
+      expect(s3).to receive(:delete_objects).with(bucket: 'testbucket', delete: {objects: [{key: 'myapp/test.tar.gz'}, {key: 'myapp/test.json'}]}).and_return(s3_delete_response)
       remove
     end
 
@@ -265,7 +266,7 @@ describe Mamiya::Storages::S3 do
       let(:package_name) { 'test.tar.gz' }
 
       it "removes specified package from S3" do
-        expect(s3).to receive(:delete_objects).with(bucket: 'testbucket', delete: {objects: [{key: 'myapp/test.tar.gz'}, {key: 'myapp/test.json'}]})
+        expect(s3).to receive(:delete_objects).with(bucket: 'testbucket', delete: {objects: [{key: 'myapp/test.tar.gz'}, {key: 'myapp/test.json'}]}).and_return(s3_delete_response)
         remove
       end
     end
@@ -274,7 +275,7 @@ describe Mamiya::Storages::S3 do
       let(:package_name) { 'test.json' }
 
       it "removes specified package from S3" do
-        expect(s3).to receive(:delete_objects).with(bucket: 'testbucket', delete: {objects: [{key: 'myapp/test.tar.gz'}, {key: 'myapp/test.json'}]})
+        expect(s3).to receive(:delete_objects).with(bucket: 'testbucket', delete: {objects: [{key: 'myapp/test.tar.gz'}, {key: 'myapp/test.json'}]}).and_return(s3_delete_response)
         remove
       end
     end
@@ -288,6 +289,21 @@ describe Mamiya::Storages::S3 do
         expect { storage.remove('test') }.to raise_error(Mamiya::Storages::Abstract::NotFound)
       end
     end
+
+    context "when delete_objects fails" do
+      let(:errors) do
+        [
+          double('Aws::S3::Errors::InteranlError', code: 'InteranlError', message: 'Internal Error', key: 'myapp/test.tar.gz'),
+          double('Aws::S3::Errors::AccessDenied', code: 'AccessDenied', message: 'Access Denied', key: 'myapp/test.json'),
+        ]
+      end
+      let(:s3_delete_response) { double('response for POST /?delete', errors: errors) }
+
+      it "raises error" do
+        expect(s3).to receive(:delete_objects).with(bucket: 'testbucket', delete: {objects: [{key: 'myapp/test.tar.gz'}, {key: 'myapp/test.json'}]}).and_return(s3_delete_response)
+        expect { remove }.to raise_error(Mamiya::Storages::S3::MultipleObjectsDeletionError, /Access Denied/)
+      end
+    end
   end
 
   describe ".find" do

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: mamiya
 version: !ruby/object:Gem::Version
-  version: 0.2.1
+  version: 0.2.2
 platform: ruby
 authors:
 - Shota Fukumori (sora_h)
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-10-20 00:00:00.000000000 Z
+date: 2014-12-10 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: thor
@@ -183,6 +183,7 @@ files:
 - docs/internal/serf_events.md
 - docs/internal/serf_queries.md
 - docs/internal/tasks.md
+- docs/quick_start.md
 - docs/upgrading.md
 - example/.gitignore
 - example/Procfile
@@ -297,7 +298,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.2.2
+rubygems_version: 2.4.1
 signing_key: 
 specification_version: 4
 summary: Fast deploy tool using tarballs and serf