cloud-utils 1.0

data/CHANGELOG

@@ -0,0 +1 @@
+v1.0. Initial release.

data/LICENSE

@@ -0,0 +1,15 @@
+#
+# Copyright (c) 2010 by J. Brisbin <jon@jbrisbin.com>
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+# or implied. See the License for the specific language governing
+# permissions and limitations under the License.
+#

data/Manifest

@@ -0,0 +1,14 @@
+CHANGELOG
+LICENSE
+README.markdown
+Rakefile
+bin/deploy
+bin/monitor
+cloud-utils.tmproj
+etc/deploy.yml
+etc/monitor.yml
+lib/cloud/deploy.rb
+lib/cloud/deploy/artifact.rb
+lib/cloud/deploy/monitor.rb
+lib/cloud/logger.rb
+Manifest

data/README.markdown

@@ -0,0 +1,53 @@
+Ruby Cloud Artifact Deployer
+=====
+
+The purpose of this utility is to aid the developer in deploying artifacts into a hybrid cloud environment. If you have several instances of an application server running on different virtual machines, then you can use these utilities to facilitate that deployment. It respects ETags and won't download a resource it's already deployed before (unless you give it the "-f" options, which forces a deployment). It also respects MD5 sums and won't redeploy a resource it's downloaded if it has already deployed it before.
+
+Config Files
+-----
+
+In order to be a little more secure, this system doesn't deploy arbitrary artifacts. You have to explicitly configure what you want downloaded and where you want it to go. There are two main configuration files: the "monitor.yml" file, which controls how the monitor watches for deployment notifications, and the "deploy.yml" file, which configures the deployer. They are both YAML files. An example config for the deployment monitor would be:
+
+<pre><code>
+default:
+  host: localhost
+  port: 5672
+  user: guest
+  password: guest
+  virtual_host: /
+  exchange: vcloud.deployment.events
+
+myapp.war:
+  :deploy: deploy -e %s
+</code></pre>
+
+The "default" section handles the RabbitMQ connection info, as well as what exchange the artifacts will be publishing deployment notifications to (more on that when I get that part uploaded). Every other section is considered to be a deployment artifact. The name should correspond to the queue name being used to publish the notifications. In this example, your continuous integration server would publish a message with the MD5 hash of the file being deployed as the "correlation_id" and the artifact name (as configured in deploy.yml) as the body of the message. The "%s" in the deploy line is where the artifact name that comes from the AMQP message. That's optional. You could specify the artifact name (as configured in deploy.yml).
+
+The deploy.yml file configures the deployer:
+
+<pre><code>
+myapp.war:
+  source: http://development/artifacts/myapp.war
+  destination:
+    - /opt/tcServer/tcServer-6.0/tc1/webapps
+    - /opt/tcServer/tcServer-6.0/tc2/webapps
+    - /opt/tcServer/tcServer-6.0/tc3/webapps
+
+myapp.html:
+  unzip: true
+  source: http://development/artifacts/myapp-html.tar.gz
+  destination:
+    - /var/www/www.mycloud.mycompany.com/public
+</code></pre>
+
+Invoking the Deployment Mechanism
+-----
+
+To deploy a file (maybe, depending on MD5 sums and ETags), you first run the "monitor" script, which checks whether or not anything needs to be deployed. It will then execute whatever is defined in the YAML file's ":deploy" section. In this case, we're calling the other Ruby script in this package: "deploy", passing it the "-e" switch, which tells it to respect ETags and not download anything if there has been no change since the last deployment, and using the "%s" replacement character, where the artifact name will be passed when the message is processed.
+
+When the deploy script is called by the monitor script, it downloads the artifact from the URL you configured in deploy.yml and either copies it directly to the list of destinations you give it, or unzips the file (if the artifact ends in ".tar.gz", it will be untarred, otherwise it will be unzipped) if you have configured "unzip: true".
+
+License
+-----
+
+Apache 2.0 licensed. Just like the other cloud utilities.

data/Rakefile

@@ -0,0 +1,9 @@
+require 'echoe'
+
+Echoe.new('cloud-utils') do |pkg|
+	pkg.author = 'J. Brisbin'
+	pkg.email = 'jon@jbrisbin.com'
+	pkg.summary = 'Deploy artifacts in a cloud environment.'
+	pkg.url = 'http://github.com/jbrisbin/cloud-utils-deployer'
+	pkg.runtime_dependencies = ['amqp >=0.6.7']
+end

data/bin/deploy

@@ -0,0 +1,50 @@
+#!/usr/bin/env ruby
+$:.unshift File.expand_path("../../lib", __FILE__)
+# puts "#{$:}"
+
+require "optparse"
+require "yaml"
+require "cloud/logger"
+require "cloud/deploy/artifact"
+
+# Set command-line options
+options = {}
+optparse = OptionParser.new do |opts|
+  opts.banner = "Usage: deploy [options] artifact1 [artifact2] [...]"
+
+  options[:config] = '/etc/cloud/deploy.yml'
+  opts.on('-c', '--config CONFIG', 'Config file to read') do |config|
+    options[:config] = config
+  end
+
+  options[:etags] = false
+  opts.on('-e', '--etags', 'Respect ETag headers when downloading resources.') do
+    options[:etags] = true
+  end
+
+  options[:force] = false
+  opts.on('-f', '--force', 'Download and deploy this file irregardless of ETags and MD5 sums.') do
+    options[:force] = true
+  end
+
+  opts.on('-h', '--help', 'Display this help') do
+    puts opts
+    exit
+  end
+end
+optparse.parse!
+
+config = YAML.load_file(options[:config])
+
+ARGV.each do |arg|
+  artifact = Cloud::Deploy::Artifact.new
+  artifact.name = arg
+  artifact.uri = config[arg]['source']
+  artifact.local_paths = config[arg]['destination']
+  artifact.unzip = config[arg]['unzip']
+  artifact.force = options[:force]
+  artifact.use_etags = options[:etags]
+  if artifact.outdated?
+    artifact.deploy!
+  end
+end

data/bin/monitor

@@ -0,0 +1,76 @@
+#!/usr/bin/env ruby
+$:.unshift File.expand_path("../../lib", __FILE__)
+
+require "optparse"
+require "yaml"
+require "cloud/logger"
+require "cloud/deploy"
+require "cloud/deploy/artifact"
+require "cloud/deploy/monitor"
+require "rubygems"
+require "mq"
+
+# Set command-line options
+options = {}
+optparse = OptionParser.new do |opts|
+  opts.banner = "Usage: monitor [options] [artifact1] [artifact...]"
+
+  options[:config] = '/etc/cloud/monitor.yml'
+  opts.on('-c', '--config CONFIG', 'Config file to read') do |config|
+    options[:config] = config
+  end
+
+  options[:force] = false
+  opts.on('-f', '--force', 'Disregard MD5 sums and force deployment') do
+    options[:force] = true
+  end
+
+  options[:timeout] = 5
+  opts.on('-t', '--timeout TIMEOUT', 'How many seconds to wait for subscriber threads to capture all messages') do |timeout|
+    options[:timeout] = timeout
+  end
+
+  opts.on('-h', '--help', 'Display this help') do
+    puts opts
+    exit
+  end
+end
+optparse.parse!
+
+config = YAML.load_file(options[:config])
+em_thread = Thread.new { EM.run }
+
+# Use the name coming in on the command line or all of the configured names
+names = Hash.new
+if ARGV.size > 0
+	$log.info('deployer') { "Was passed artifacts to deploy, so using those: #{ARGV.join(', ')}" }
+	names.replace(ARGV)
+else
+	$log.info('deployer') { "Deploying all configured artifacts" }
+	names.replace(config)
+	names.delete_if { |k, v| k == "default" }
+end
+
+monitors = []
+names.each do |k, v|
+	mon = Cloud::Deploy::Monitor.new(config, k)
+	mon.deploy!
+	monitors << mon
+end
+em_thread.join(options[:timeout])
+
+# Finish processing
+Cloud::Deploy.load_md5sums do |md5s|
+	monitors.each do |mon|
+		if !mon.artifact.nil?
+			deploy_cmd = config[mon.name][:deploy] % mon.artifact
+			already_seen = md5s[mon.md5sum].nil?
+			if !options[:force] and already_seen
+				$log.info(mon.name) { "MD5 sums match, so not deploying again" }
+			else
+				$log.info(mon.name) { "Deploying: #{mon.artifact} with hash: #{mon.md5sum} using: #{deploy_cmd}" }
+				`#{deploy_cmd}`
+			end
+		end
+	end
+end

data/cloud-utils.gemspec

@@ -0,0 +1,34 @@
+# -*- encoding: utf-8 -*-
+
+Gem::Specification.new do |s|
+  s.name = %q{cloud-utils}
+  s.version = "1.0"
+
+  s.required_rubygems_version = Gem::Requirement.new(">= 1.2") if s.respond_to? :required_rubygems_version=
+  s.authors = ["J. Brisbin"]
+  s.date = %q{2010-06-14}
+  s.description = %q{Deploy artifacts in a cloud environment.}
+  s.email = %q{jon@jbrisbin.com}
+  s.executables = ["deploy", "monitor"]
+  s.extra_rdoc_files = ["CHANGELOG", "LICENSE", "README.markdown", "bin/deploy", "bin/monitor", "lib/cloud/deploy.rb", "lib/cloud/deploy/artifact.rb", "lib/cloud/deploy/monitor.rb", "lib/cloud/logger.rb"]
+  s.files = ["CHANGELOG", "LICENSE", "README.markdown", "Rakefile", "bin/deploy", "bin/monitor", "cloud-utils.tmproj", "etc/deploy.yml", "etc/monitor.yml", "lib/cloud/deploy.rb", "lib/cloud/deploy/artifact.rb", "lib/cloud/deploy/monitor.rb", "lib/cloud/logger.rb", "Manifest", "cloud-utils.gemspec"]
+  s.homepage = %q{http://github.com/jbrisbin/cloud-utils-deployer}
+  s.rdoc_options = ["--line-numbers", "--inline-source", "--title", "Cloud-utils", "--main", "README.markdown"]
+  s.require_paths = ["lib"]
+  s.rubyforge_project = %q{cloud-utils}
+  s.rubygems_version = %q{1.3.7}
+  s.summary = %q{Deploy artifacts in a cloud environment.}
+
+  if s.respond_to? :specification_version then
+    current_version = Gem::Specification::CURRENT_SPECIFICATION_VERSION
+    s.specification_version = 3
+
+    if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
+      s.add_runtime_dependency(%q<amqp>, [">= 0.6.7"])
+    else
+      s.add_dependency(%q<amqp>, [">= 0.6.7"])
+    end
+  else
+    s.add_dependency(%q<amqp>, [">= 0.6.7"])
+  end
+end

data/cloud-utils.tmproj

@@ -0,0 +1,48 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+	<key>currentDocument</key>
+	<string>lib/cloud/deploy/artifact.rb</string>
+	<key>documents</key>
+	<array>
+		<dict>
+			<key>expanded</key>
+			<true/>
+			<key>name</key>
+			<string>cloud-utils</string>
+			<key>regexFolderFilter</key>
+			<string>!.*/(\.[^/]*|CVS|_darcs|_MTN|\{arch\}|blib|.*~\.nib|.*\.(framework|app|pbproj|pbxproj|xcode(proj)?|bundle))$</string>
+			<key>sourceDirectory</key>
+			<string></string>
+		</dict>
+	</array>
+	<key>fileHierarchyDrawerWidth</key>
+	<integer>260</integer>
+	<key>metaData</key>
+	<dict>
+		<key>lib/cloud/deploy/artifact.rb</key>
+		<dict>
+			<key>caret</key>
+			<dict>
+				<key>column</key>
+				<integer>61</integer>
+				<key>line</key>
+				<integer>125</integer>
+			</dict>
+			<key>firstVisibleColumn</key>
+			<integer>0</integer>
+			<key>firstVisibleLine</key>
+			<integer>0</integer>
+		</dict>
+	</dict>
+	<key>openDocuments</key>
+	<array>
+		<string>lib/cloud/deploy/artifact.rb</string>
+	</array>
+	<key>showFileHierarchyDrawer</key>
+	<true/>
+	<key>windowFrame</key>
+	<string>{{270, 4}, {1410, 1024}}</string>
+</dict>
+</plist>

data/etc/deploy.yml

@@ -0,0 +1,12 @@
+myapp.war:
+  source: http://development/artifacts/myapp.war
+  destination:
+    - /opt/tcServer/tcServer-6.0/tc1/webapps
+    - /opt/tcServer/tcServer-6.0/tc2/webapps
+    - /opt/tcServer/tcServer-6.0/tc3/webapps
+
+myapp.html:
+  unzip: true
+  source: http://development/artifacts/myapp-html.tar.gz
+  destination:
+    - /var/www/www.mycloud.mycompany.com/public

data/etc/monitor.yml

@@ -0,0 +1,10 @@
+default:
+  host: localhost
+  port: 5672
+  user: guest
+  password: guest
+  virtual_host: /
+  exchange: vcloud.deployment.events
+
+myapp.war:
+  :deploy: deploy -e %s

data/lib/cloud/deploy.rb

@@ -0,0 +1,63 @@
+#
+# Copyright (c) 2010 by J. Brisbin <jon@jbrisbin.com>
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+# or implied. See the License for the specific language governing
+# permissions and limitations under the License.
+#
+require "cloud/logger"
+
+module Cloud
+	module Deploy
+		
+		ETAG_FILE = '/var/lib/cloud/deploy.etags'
+		MD5SUM_FILE = '/var/lib/cloud/deploy.md5sums'
+		
+		def load_etags
+      etags = {}
+      if @use_etags
+        begin
+          File.open(ETAG_FILE, 'r') do |f|
+            etags = Marshal.load(f)
+          end
+        rescue => err
+          $log.fatal('etags') { err }
+        end
+      end
+      yield etags
+    end
+
+    def save_etags(etags)
+      File.open(ETAG_FILE, 'w') do |f|
+        f.write(Marshal.dump(etags))
+      end
+    end
+
+    def load_md5sums
+      md5sums = {}
+      begin
+        File.open(MD5SUM_FILE, 'r') do |f|
+          md5sums = Marshal.load(f)
+        end
+      rescue => err
+        $log.fatal('md5sums') { err }
+      end
+      yield md5sums
+    end
+
+    def save_md5sums(md5sums)
+      File.open(MD5SUM_FILE, 'w') do |f|
+        f.write(Marshal.dump(md5sums))
+      end
+    end
+    
+	end
+end

data/lib/cloud/deploy/artifact.rb

@@ -0,0 +1,124 @@
+#
+# Copyright (c) 2010 by J. Brisbin <jon@jbrisbin.com>
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+# or implied. See the License for the specific language governing
+# permissions and limitations under the License.
+#
+require "cloud/logger"
+require "cloud/deploy"
+require "net/http"
+require "uri"
+require "fileutils"
+include Cloud::Deploy
+
+module Cloud
+  module Deploy
+
+    class Artifact
+
+      attr_reader :uri
+      attr_accessor :name, :local_paths, :unzip, :force, :use_etags
+
+      def initialize
+        @use_etags = true
+      end
+
+      def uri=(remote_uri)
+        @uri = URI.parse(remote_uri)
+        @http = Net::HTTP.new(@uri.host, @uri.port)
+      end
+
+      def outdated?
+        download
+      end
+
+      def deploy!
+        load_md5sums do |md5sums|
+          hash = md5sums[@name]
+          if !@force and !hash.nil? and hash == @hash
+            $log.info(@name) { "MD5 sums match. No need to deploy." }
+          else
+            @local_paths.each do |path|
+              FileUtils.mkdir_p(path)
+              if @unzip
+                $log.info(@name) { "Unzipping #{@name} to #{path}" }
+                if @uri.request_uri[-6..-1] == "tar.gz"
+                  `tar -zxf #{@temp_file} -C #{path}`
+                else
+                  `unzip -d #{path} #{@temp_file}`
+                end
+              else
+                $log.info(@name) { "Copying #{@name} to #{path}" }
+                FileUtils.cp(@temp_file, path)
+              end
+            end
+            md5sums[@name] = @hash
+            save_md5sums(md5sums)
+          end
+          # Clean up temp file
+          if !@temp_file.nil?
+            FileUtils.rm(@temp_file)
+          end
+          # We did a deployment
+          return true
+        end
+        # Default return
+        return false
+      end
+
+      private
+      def download
+        outdated = false
+
+        request = Net::HTTP::Get.new(@uri.request_uri)
+        load_etags do |etags|
+          etag = etags[@name]
+          if !@force and !etag.nil?
+            request.initialize_http_header({
+                'If-None-Match' => etag
+            })
+          end
+
+          response = @http.request(request)
+          case response
+            when Net::HTTPSuccess
+              # Continue to download file...
+              $log.info(@name) { "Downloading: #{@uri.to_s}..." }
+              bytes = response.body
+              require "md5"
+              @hash = MD5.new.update(bytes).hexdigest
+              # Write to temp file, ready to deploy
+              @temp_file = "/tmp/#{@name}"
+              File.open(@temp_file, "w") { |f| f.write(bytes) }
+              # Update ETags
+              etags[@name] = response['etag']
+
+              outdated = true
+            when Net::HTTPNotModified
+              # No need to download it again
+              $log.info(@name) { "ETag matched, not downloading: #{@uri.to_s}" }
+            else
+              $log.fatal(@name) { "Error HTTP status code received: #{response['code']}" }
+          end
+
+          if @use_etags
+            save_etags(etags)
+          end
+        end
+
+        return outdated
+      end
+
+    end
+
+  end
+end

data/lib/cloud/deploy/monitor.rb

@@ -0,0 +1,57 @@
+#
+# Copyright (c) 2010 by J. Brisbin <jon@jbrisbin.com>
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+# or implied. See the License for the specific language governing
+# permissions and limitations under the License.
+#
+require "logger"
+require "rubygems"
+require "mq"
+require "cloud/logger"
+require "cloud/deploy"
+
+module Cloud
+  module Deploy
+    class Monitor
+
+			attr_reader :md5sum, :artifact
+      attr_accessor :config, :name
+
+			def initialize(config, name)
+				@config = config
+				@name = name
+				@exchange = @config['default']['exchange']
+			end
+
+      def deploy!
+				defaults = @config['default']
+
+				# Connect to RabbitMQ...
+				host = defaults['host']
+				port = defaults['port']
+				user = defaults['user']
+				pass = defaults['password']
+				vhost = defaults['virtual_host']
+				exchange = defaults['exchange']
+				$log.info(@name) { "Connecting to MQ h: #{host}, p: #{port}, u: #{user}, pw: #{pass}, v: #{vhost}, x: #{exchange}" }
+
+				AMQP.start(:host => host, :port => port, :user => user, :pass => pass, :vhost => vhost) do
+	        MQ.queue(@name).bind(@exchange).subscribe(:ack => true) do |headers, body|
+						@md5sum = headers.properties[:correlation_id]
+						@artifact = body
+					end
+				end
+      end
+
+    end
+  end
+end

data/lib/cloud/logger.rb

@@ -0,0 +1,18 @@
+#
+# Copyright (c) 2010 by J. Brisbin <jon@jbrisbin.com>
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+# or implied. See the License for the specific language governing
+# permissions and limitations under the License.
+#
+require "logger"
+
+$log = Logger.new('/var/log/cloud/deployer.log')

metadata

@@ -0,0 +1,110 @@
+--- !ruby/object:Gem::Specification 
+name: cloud-utils
+version: !ruby/object:Gem::Version 
+  hash: 15
+  prerelease: false
+  segments: 
+  - 1
+  - 0
+  version: "1.0"
+platform: ruby
+authors: 
+- J. Brisbin
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2010-06-14 00:00:00 -05:00
+default_executable: 
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: amqp
+  prerelease: false
+  requirement: &id001 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        hash: 9
+        segments: 
+        - 0
+        - 6
+        - 7
+        version: 0.6.7
+  type: :runtime
+  version_requirements: *id001
+description: Deploy artifacts in a cloud environment.
+email: jon@jbrisbin.com
+executables: 
+- deploy
+- monitor
+extensions: []
+
+extra_rdoc_files: 
+- CHANGELOG
+- LICENSE
+- README.markdown
+- bin/deploy
+- bin/monitor
+- lib/cloud/deploy.rb
+- lib/cloud/deploy/artifact.rb
+- lib/cloud/deploy/monitor.rb
+- lib/cloud/logger.rb
+files: 
+- CHANGELOG
+- LICENSE
+- README.markdown
+- Rakefile
+- bin/deploy
+- bin/monitor
+- cloud-utils.tmproj
+- etc/deploy.yml
+- etc/monitor.yml
+- lib/cloud/deploy.rb
+- lib/cloud/deploy/artifact.rb
+- lib/cloud/deploy/monitor.rb
+- lib/cloud/logger.rb
+- Manifest
+- cloud-utils.gemspec
+has_rdoc: true
+homepage: http://github.com/jbrisbin/cloud-utils-deployer
+licenses: []
+
+post_install_message: 
+rdoc_options: 
+- --line-numbers
+- --inline-source
+- --title
+- Cloud-utils
+- --main
+- README.markdown
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      hash: 3
+      segments: 
+      - 0
+      version: "0"
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      hash: 11
+      segments: 
+      - 1
+      - 2
+      version: "1.2"
+requirements: []
+
+rubyforge_project: cloud-utils
+rubygems_version: 1.3.7
+signing_key: 
+specification_version: 3
+summary: Deploy artifacts in a cloud environment.
+test_files: []
+