whiskey_disk 0.6.4 → 0.6.10

data/CHANGELOG

@@ -1,4 +1,38 @@
 
+0.6.10 / 2011-02-08 
+==================
+
+  * bump version to 0.6.10
+  * Merge branch 'feature/improved-integration-specs' into develop
+  * reorganizing integration specs
+  * adding a quick README on how to run integration specs
+  * adding a remote deployment integration spec
+  * working around permissioning issues to support "remote" integration specs
+  * integration specs for non-matching --only deployment
+  * make sure the integration test is actually testing the local library
+  * fixing 'unit@' --only local deployment WD.remote? bug
+  * Adding initial integration spec suite
+  * save +x chmod state to bin/wd
+  * Merge branch 'develop' of git.ogtastic.com:whiskey_disk into develop
+  * Merge branch 'hotfix/cd-before-post-scripts' into develop
+  * adding --only support to wd binary
+  * an ENV['only'] domain should deploy locally
+  * adding support for ENV['none'] to WhiskeyDisk, ::Config
+  * updating README to document upcoming --only / ENV['only'] functionality
+  * WD.fetch now iterates over all domains, local or remote
+  * reorganizing specs
+  * make it an error for a domain to appear more than once in a given project/target
+  * changing internal :domain representation
+  * WhiskeyDisk.remote? now takes a domain argument
+  * README updates for specifying local deployments via "local" domain
+  * added tl;dr quickstart section to README
+  * Merge branch 'feature/enable_pulling_deploy.yml_from_url' into develop
+  * updating README to describe using --path URLs
+  * can now specify path to deploy.yml info using an URL
+  * Merge branch 'feature/move_todo_to_pivotal_tracker' into develop
+  * removing TODO.txt
+  * updating README to point to PT project instead of old TODO.txt file
+
 0.6.4 / 2011-01-26 
 ==================
 

data/README.integration_specs

@@ -0,0 +1,36 @@
+It is possible to run an integration spec suite which attempts to exercise the
+full whiskey_disk stack.  I want this to be able to work on a single machine
+(especially since I like to work on whiskey_disk on airplanes and in foreign 
+countries with questionable bandwidth/infrastructure).  So, here's how I set my
+environment up to make this possible
+
+
+     - need the following /etc/hosts entry:
+       127.0.0.1       localhost wd-git.example.com wd-app1.example.com wd-app2.example.com
+
+     - enable sshd logins
+     
+     - create a local user named 'user', then:
+     
+     user% mkdir wd-integration-target
+     user% chmod 777 wd-integration-target
+     user% ln -s ~/wd-integration-target /tmp/wd-integration-target
+     
+     user% ssh-keygen  (specify no passphrase)
+     user% cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
+     user% chmod 700 ~/.ssh/; chmod 600 ~/.ssh/authorized_keys
+     
+     user% ssh user@wd-app1.example.com  (answer 'yes' to the trust host prompt)
+     ^D
+     user% ssh user@wd-app2.example.com  ( ditto )
+     ^D
+
+     then, as user, git clone whiskey_disk from github or ln -s to an already checked out version
+     (which is what I usually do)
+     
+     - run git-daemon:
+       user% cd git/whiskey_disk
+       user% git daemon --base-path=`pwd`/scenarios/git_repositories/ --reuseaddr --verbose &
+
+     user% export INTEGRATION=true
+     user% rake   

data/README.markdown

@@ -8,6 +8,32 @@ Right-arrow through a short whiskey_disk presentation at [http://wd2010.rickbrad
 You can also right-arrow through a shorter but more up-to-date whiskey_disk "lightning talk" presentation 
 (from the 2010 Ruby Hoedown) at [http://wdlightning.rickbradley.com/](http://wdlightning.rickbradley.com) (slide source available [here](http://github.com/rick/whiskey_disk_presentation/tree/lightning).), covering the 0.4.*-era functionality.
 
+### tl;dr ###
+
+First:
+
+    % gem install whiskey_disk
+
+
+Then make a deploy.yml file (in config/ if you're doing a Rails project):
+
+    staging:
+      domain: "deployment_user@staging.mydomain.com"
+      deploy_to: "/path/to/where/i/deploy/staging.mydomain.com"
+      repository: "https://github.com/username/project.git"
+      branch: "staging"
+      rake_env:
+        RAILS_ENV: 'production'	    
+
+then:
+
+    % wd setup --to=staging
+
+then:
+
+    % wd deploy --to=staging
+
+
 ### Selling points ###
 
   - If you share the same opinions as we do there's almost no code involved, almost no
@@ -142,6 +168,26 @@ of deploy:setup
    lib/tasks/ or in your project's Rakefile) will cause that task to be run
 at the end of deploy:now
 
+It's easy to specify a local deployment.  The simplest way is to just not specify a "domain":
+
+    local:
+      deploy_to: "/var/www/www.ogtastic.com"
+      repository: "git@ogtastic.com:www.ogtastic.com.git"
+      branch: "stable"
+      rake_env:
+        RAILS_ENV: 'production'
+
+
+Or, just specify the string 'local' as the domain:
+
+    local:
+      domain: "local"
+      deploy_to: "/var/www/www.ogtastic.com"
+      repository: "git@ogtastic.com:www.ogtastic.com.git"
+      branch: "stable"
+      rake_env:
+        RAILS_ENV: 'production'
+
 
 For deploying to multiple hosts, the config/deploy.yml might look like:
 
@@ -156,6 +202,19 @@ For deploying to multiple hosts, the config/deploy.yml might look like:
         RAILS_ENV: 'production'
 
 
+You can even include a local deployment along with remote deployments, simply use the 'local' name:
+
+    qa:
+      domain:
+      - "local"
+      - "ogc@qa2.ogtastic.com""
+      deploy_to: "/var/www/www.ogtastic.com"
+      repository: "git@ogtastic.com:www.ogtastic.com.git"
+      branch: "stable"
+      rake_env:
+        RAILS_ENV: 'production'
+
+
 ### Specifying domains, with or without roles ###
 
 There are a number of ways to specify domains (the ssh connection strings denoting the hosts
@@ -187,7 +246,6 @@ otherwise it's optional and superfluous):
       domain:
       - name: "build@ci.example.com"
 
-
 It's also possible to assign various "roles" to the domains to which you deploy.  Some common usages would be
 "www", which might need a post\_deploy task which notifies some web server software (apache, nginx, passenger, 
 unicorn, etc.) that it should refresh the contents being served; or perhaps "db", which might need some set of
@@ -231,6 +289,17 @@ But domains with roles can be specified alongside simple domains as well:
         - "db"
 
 
+And, if you need to assign roles for a local deployment, you can do that as well:
+
+    local:
+      domain:
+      - name: "local"
+        roles: 
+        - "web"
+        - "app"
+        - "db"
+
+
 All that said, it's often simpler to refrain from carving up hosts into roles.  But who's going to listen to reason?
 
 
@@ -364,18 +433,44 @@ are in an environment where the 'app' role is active but the 'web' role is not:
     % wd setup --to=<target>
     % wd setup --to=<project>:<target>
     % wd setup --to=foo:qa --path=/etc/whiskey_disk/deploy.yml
-    
+    % wd setup --to=foo:qa --path=https://github.com/username/project/raw/master/path/to/configs/deploy.yml
+    % wd setup --to=foo:qa --only=myhost.example.com
+
     % wd deploy --to=<target>
     % wd deploy --to=<project>:<target>
     % wd deploy --to=foo:qa --path=/etc/whiskey_disk/deploy.yml
-
+    % wd deploy --to=foo:qa --path=https://github.com/username/project/raw/master/path/to/configs/deploy.yml
+    % wd deploy --to=foo:qa --only=myhost.example.com
 
 Note that the wd command (unlike rake, which requires a Rakefile in the current directory) can be run from anywhere, so you can deploy any project, working from any path, and can even specify where to find the deployment YAML configuration file.
   
 The --path argument can take either a file or a directory.  When given a file it will use that file as the configuration file.  When given a directory it will look in that directory for deploy/&lt;project&gt;/&lt;target&gt;.yml, then deploy/&lt;project&gt;.yml, then deploy/&lt;target&gt;.yml, then &lt;target&gt;.yml, and finally, deploy.yml.
-  
+
+To make things even better, you can provide an URL as the --path argument and have a central location from which to pull deployment YAML data.  This means that you can centrally administer the definitive deployment information for the various projects and targets you manage.  This could be as simple as keeping them in a text file hosted on a web server, checking them into git and using github or gitweb to serve up the file contents on HEAD, or it could be a programmatically managed configuration management system returning dynamically-generated results.
+
 All this means you can manage a large number of project deployments (local or remote) and have a single scripted deployment manager that keeps them up to date.  Configurations can live in a centralized location, and developers don't have to be actively involved in ensuring code gets shipped up to a server.  Win.
 
+When doing scripted deployments for a group of nodes who appear in the same 'domain' list, it's possible to specify the --only setting so that you can identify which domain entries belong to a specific node.  For example, given this configuration:
+
+
+    production:
+      domain:
+      - foo.example.com
+      - bar.example.com
+      repository: git@github.com:foo.git
+
+
+We would like to be able to set up the following scripted runs on foo.example.com and bar.example.com:
+
+
+    foo% wd deploy --to=app:production --path=http://automation.example.com/wd/deploy.yml
+    bar% wd deploy --to=app:production --path=http://automation.example.com/wd/deploy.yml
+
+
+But without specifying --only we end up with undesired results.  When foo.example.com runs it will see that it needs to make sure deployment happens on 'foo.example.com' and 'bar.example.com'.  So, wd will ssh from foo.example.com to foo.example.com (less than ideal), deploy, and then ... it will ssh from foo.example.com to bar.example.com and do a deployment.  When bar.example.com runs, however, it will also ssh to bar.example.com, and also to foo.example.com.  So each host will be deployed twice.
+
+The --only setting is used to tell a node what its name is, and to tell it not to bother trying to deploy other nodes that it finds in the target's "domain" listing.  In other words, deployment is being managed from afar, and it's best to just manage ourselves and forego managing other nodes.
+
 
 ### A note about post\_{setup,deploy} Rake tasks
 
@@ -403,6 +498,28 @@ In your Rakefile:
     % rake deploy:setup to=<project>:<target>   (e.g., "foo:qa", "bar:production", etc.)
     % rake deploy:now   to=<project>:<target>
 
+  enabling staleness checking (see below):
+
+    % rake deploy:setup to=<project>:<target> check=yes
+    % rake deploy:now   to=<project>:<target> check=yes
+
+  maybe even specifying the path to the configuration file:
+
+    % rake deploy:setup to=<project>:<target> path=/etc/deploy.yml
+    % rake deploy:now   to=<project>:<target> path=/etc/deploy.yml
+
+  how about specifying the configuration file via URL:
+
+    % rake deploy:setup to=<project>:<target> path=https://github.com/username/project/raw/master/path/to/configs/deploy.yml
+    % rake deploy:now   to=<project>:<target> path=https://github.com/username/project/raw/master/path/to/configs/deploy.yml
+
+  Finally, it's also possible to specify the 'only' variable to limit 'domain' entries of interest:
+
+    % rake deploy:setup to=<project>:<target> only=myhost.example.com
+    % rake deploy:now to=<project>:<target> only=myhost.example.com
+
+  (see the discussion of --only above in "Running whiskey\_disk from the command-line" for more information)
+
 
 ### Staleness checks ###
 
@@ -637,7 +754,7 @@ Notice that there are no separate trees for 'uat' and 'qa' targets.
 
 ### Future Directions ###
 
-Check out the [TODO.txt](http://github.com/flogic/whiskey_disk/raw/master/TODO.txt) file 
+Check out the [Pivotal Tracker project](https://www.pivotaltracker.com/projects/202125)
 to see what we have in mind for the near future.
 
 ### Resources ###

data/VERSION

@@ -1 +1 @@
-0.6.4
+0.6.10

data/bin/wd

@@ -15,6 +15,10 @@ op = OptionParser.new do |opts|
     options[:path] = path
   end
     
+  opts.on('-o=DOMAIN', '--only=DOMAIN', "limit deployment to this domain") do |domain|
+    options[:only] = domain
+  end
+    
   opts.on('-c', '--check', "do a staleness check before deploying") do |path|
     options[:check] = 'true'
   end
@@ -32,6 +36,7 @@ raise op.to_s unless ['deploy', 'setup'].include?(command)
 
 ENV['to'] = options[:target]
 ENV['path'] = options[:path]
+ENV['only'] = options[:only]
 ENV['check'] = options[:check]
 
 if command == 'deploy'

data/lib/whiskey_disk/config.rb

@@ -1,4 +1,6 @@
 require 'yaml'
+require 'uri'
+require 'open-uri'
 
 class WhiskeyDisk
   class Config
@@ -26,6 +28,11 @@ class WhiskeyDisk
       def check_staleness?
         !!(ENV['check'] && ENV['check'] =~ /^(?:t(?:rue)?|y(?:es)?|1)$/)
       end
+      
+      def domain_limit
+        return false unless ENV['only'] and ENV['only'] != ''
+        ENV['only']
+      end
 
       def contains_rakefile?(path)
         File.exists?(File.expand_path(File.join(path, 'Rakefile')))
@@ -47,9 +54,19 @@ class WhiskeyDisk
         find_rakefile_from_current_path
       end
 
-      def configuration_file
-        return path if path and File.file?(path)
+      def valid_path?(path)
+        if path
+          uri = URI.parse(path)
+          return path if uri.scheme
+          return path if File.file?(path)
+        end
+        
+        false
+      end
 
+      def configuration_file
+        return path if valid_path?(path)
+        
         files = []
 
         files += [
@@ -69,8 +86,7 @@ class WhiskeyDisk
       end
 
       def configuration_data
-        raise "Configuration file [#{configuration_file}] not found!" unless File.exists?(configuration_file)
-        File.read(configuration_file)
+        open(configuration_file) {|f| f.read }
       end
 
       def project_name
@@ -104,13 +120,17 @@ class WhiskeyDisk
         { project_name => data }
       end
 
+      def localize_domain_list(list)
+        [ list ].flatten.collect { |d| (d.nil? or d == '') ? 'local' : d }
+      end
+      
       def compact_list(list)
         [ list ].flatten.delete_if { |d| d.nil? or d == '' }
       end
       
       def normalize_domain(data)
-        compacted = compact_list(data)
-        return nil if compacted.empty?
+        compacted = localize_domain_list(data)
+        compacted = [ 'local' ] if compacted.empty?
         
         compacted.collect do |d|
           if d.respond_to?(:keys)
@@ -124,10 +144,18 @@ class WhiskeyDisk
         end
       end
       
+      def check_duplicates(project, target, domain_list)
+        seen = {}
+        domain_list.each do |domain|
+          raise "duplicate domain [#{domain[:name]}] in configuration file for project [#{project}], target [#{target}]" if seen[domain[:name]]
+          seen[domain[:name]] = true
+        end
+      end
+      
       def normalize_domains(data)
         data.each_pair do |project, project_data|
           project_data.each_pair do |target, target_data|
-            target_data['domain'] = normalize_domain(target_data['domain']) if target_data['domain']
+            target_data['domain'] = check_duplicates(project, target, normalize_domain(target_data['domain']))
           end
         end
       end

data/lib/whiskey_disk.rb

@@ -40,8 +40,13 @@ class WhiskeyDisk
       buffer << command
     end
     
-    def remote?
-      !! self[:domain]
+    def remote?(domain)
+      return false unless domain
+      return false if domain == 'local'
+      limit = WhiskeyDisk::Config.domain_limit 
+      return false if limit and domain_limit_match?(domain, limit)
+
+      true
     end
     
     def has_config_repo?
@@ -108,25 +113,42 @@ class WhiskeyDisk
       return '' if buffer.empty?
       (staleness_checks_enabled? and check_staleness?) ? apply_staleness_check(join_commands) : join_commands
     end
+
+    def domain_limit_match?(domain, limit)
+      domain.sub(%r{^.*@}, '') == limit
+    end
+    
+    def domain_of_interest?(domain)
+      return true unless limit = WhiskeyDisk::Config.domain_limit
+      domain_limit_match?(domain, limit)
+    end
     
     def encode_roles(roles)
       return '' unless roles and !roles.empty?
       "export WD_ROLES='#{roles.join(':')}'; "
     end
+
+    def build_command(domain, cmd)
+      "set -x; " + encode_roles(domain[:roles]) + cmd
+    end
     
-    def run(cmd)
+    def run(domain, cmd)
+      system('ssh', '-v', domain[:name], build_command(domain, cmd))
+    end
+    
+    def shell(domain, cmd)
+      system(build_command(domain, cmd))
+    end
+    
+    def flush
       needs(:domain)
       self[:domain].each do |domain|
-        status = system('ssh', '-v', domain[:name], "set -x; " + encode_roles(domain[:roles]) + cmd)
+        next unless domain_of_interest?(domain[:name])
+        status = remote?(domain[:name]) ? run(domain, bundle) : shell(domain, bundle)
         record_result(domain[:name], status)
       end
     end
     
-    def flush
-      return run(bundle) if remote?
-      record_result('local', system(bundle))
-    end
-    
     def record_result(domain, status)
       @results ||= []
       @results << { :domain => domain, :status => status }

data/scenarios/git_repositories/config.git/HEAD

@@ -0,0 +1 @@
+ref: refs/heads/master

data/scenarios/git_repositories/config.git/config

@@ -0,0 +1,5 @@
+[core]
+	repositoryformatversion = 0
+	filemode = true
+	bare = true
+	ignorecase = true