whiskey_disk 0.6.22 → 0.6.23

data/CHANGELOG

@@ -1,11 +1,21 @@
 
-0.6.22 / 2011-04-10 
+0.6.23 / 2011-09-29
+==================
+
+  * Fix incompatibility with latest jeweler (Rakefile error messages) [James Cox (imajes)]
+  * We now use vagrant for integration specs.
+  * Make rsync_changes more robust
+  * Add ERB support to YAML files used in integration specs
+  * add per-domain ssh_options support to config files
+  * Big refactoring of config handling into a filter stream architecture
+
+0.6.22 / 2011-04-10
 ==================
 
   * support branch-switching on deployments
   * added integration spec for "branch switching" behavior
 
-0.6.21 / 2011-04-06 
+0.6.21 / 2011-04-06
 ==================
 
   * change rsync changes capturing mechanism to be more portable
@@ -14,7 +24,7 @@
   * deploy.yml to test rsync changes w/ config repo
   * integration spec project and config changes to stored git repos
 
-0.6.20 / 2011-04-06 
+0.6.20 / 2011-04-06
 ==================
 
   * Output the entire command before running it when debugging.
@@ -48,7 +58,7 @@
   * README updates for #changed? helper functionality
 
 
-0.6.17 / 2011-03-25 
+0.6.17 / 2011-03-25
 ==================
 
   * updating contributor info in README
@@ -76,7 +86,7 @@
   * Add support section to README
 
 
-0.6.16 / 2011-03-08 
+0.6.16 / 2011-03-08
 ==================
 
   * Turning off shallow clones for 'wd setup'
@@ -85,7 +95,7 @@
   * adding integration specs to test setup + deploy at once
   * tweaking a couple of old path specs
 
-0.6.15 / 2011-03-07 
+0.6.15 / 2011-03-07
 ==================
 
   * integration specs for "git checkout #{branch}" functionality
@@ -93,12 +103,12 @@
   * Adding current_branch integration spec helper method
   * Ensure specified branch is checked out before applying the refresh
 
-0.6.14 / 2011-02-23 
+0.6.14 / 2011-02-23
 ==================
 
   * forcing bash as the execution shell
 
-0.6.13 / 2011-02-20 
+0.6.13 / 2011-02-20
 ==================
 
   * pass rake_env settings to post_* scripts
@@ -107,7 +117,7 @@
   * basic integration specs for post_* scripts
   * updating integration spec git repo data
 
-0.6.12 / 2011-02-20 
+0.6.12 / 2011-02-20
 ==================
 
   * bad Rakefile should cause deployment to fail
@@ -115,13 +125,13 @@
   * new scenarios to use when testing post_* rake hooks
   * adding a dump_log helper to use when developing integration specs
 
-0.6.11 / 2011-02-18 
+0.6.11 / 2011-02-18
 ==================
 
   * Generate an error message when we can't find --to proj/env in a config
   * Fixing typo in an integration spec's example text
 
-0.6.10 / 2011-02-08 
+0.6.10 / 2011-02-08
 ==================
 
   * reorganizing integration specs
@@ -143,17 +153,17 @@
   * WhiskeyDisk.remote? now takes a domain argument
   * can now specify path to deploy.yml info using an URL
 
-0.6.4 / 2011-01-26 
+0.6.4 / 2011-01-26
 ==================
 
   * bugfix: cd to deploy_to path before running post_{setup,deploy}_script
 
-0.6.3 / 2011-01-13 
+0.6.3 / 2011-01-13
 ==================
 
   * adding support for config_target in deploy.yml
 
-0.6.2 / 2010-12-23 
+0.6.2 / 2010-12-23
 ==================
 
   * update multiple deployment example to use roles
@@ -168,7 +178,7 @@
   * reordering domain config spec setup data structure for readability
 
 
-0.6.0 / 2010-12-22 
+0.6.0 / 2010-12-22
 ==================
 
   * adding a set of multi-domain examples

data/README.integration_specs

@@ -1,36 +1,24 @@
-It is possible to run an integration spec suite which attempts to exercise the
-full whiskey_disk stack.  I want this to be able to work on a single machine
-(especially since I like to work on whiskey_disk on airplanes and in foreign 
-countries with questionable bandwidth/infrastructure).  So, here's how I set my
-environment up to make this possible
-
-
-     - need the following /etc/hosts entry:
-       127.0.0.1       localhost wd-git.example.com wd-app1.example.com wd-app2.example.com
-
-     - enable sshd logins
-     
-     - create a local user named 'user', then:
-     
-     user% mkdir wd-integration-target
-     user% chmod 777 wd-integration-target
-     user% ln -s ~/wd-integration-target /tmp/wd-integration-target
-     
-     user% ssh-keygen  (specify no passphrase)
-     user% cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
-     user% chmod 700 ~/.ssh/; chmod 600 ~/.ssh/authorized_keys
-     
-     user% ssh user@wd-app1.example.com  (answer 'yes' to the trust host prompt)
-     ^D
-     user% ssh user@wd-app2.example.com  ( ditto )
-     ^D
-
-     then, as user, git clone whiskey_disk from github or ln -s to an already checked out version
-     (which is what I usually do)
-     
-     - run git-daemon:
-       user% cd git/whiskey_disk
-       user% git daemon --base-path=`pwd`/scenarios/git_repositories/ --reuseaddr --verbose &
-
-     user% export INTEGRATION=true
-     user% rake   
+Running Integration Specs
+-------------------------
+
+To run the integration spec suite you need to be able to run a Vagrant virtual machine (see: http://vagrantup.com/), which means you will have to have VirtualBox installed.  The Vagrant/VirtualBox setup is currently out of the scope of this document.  By default, I have been using the lucid32 box as my "base" box.
+
+Once you are set up to be able to run Vagrant, you can create and start up an integration environment by running the following rake task:
+
+% rake integration:up
+
+This will create a vagrant VM, and so may take a few minutes to complete.
+
+
+To run the integration specs once the integration environment is up:
+
+    % export INTEGRATION=true
+    % rake
+
+To shut down the integration VM and the integration git-daemon server:
+
+    % rake integration:down
+
+To completely remove the integration spec VM (which can take up a good bit of space on disk), do:
+
+    % rake integration:destroy

data/README.markdown

@@ -5,7 +5,7 @@ A very opinionated deployment tool, designed to be as fast as technologically po
 
 Right-arrow through a short whiskey_disk presentation at [http://wd2010.rickbradley.com/](http://wd2010.rickbradley.com) (slide source available [here](http://github.com/rick/whiskey_disk_presentation).), covering the 0.2.*-era functionality.
 
-You can also right-arrow through a shorter but more up-to-date whiskey_disk "lightning talk" presentation 
+You can also right-arrow through a shorter but more up-to-date whiskey_disk "lightning talk" presentation
 (from the 2010 Ruby Hoedown) at [http://wdlightning.rickbradley.com/](http://wdlightning.rickbradley.com) (slide source available [here](http://github.com/rick/whiskey_disk_presentation/tree/lightning).), covering the 0.4.*-era functionality.
 
 ### tl;dr ###
@@ -23,7 +23,7 @@ Then make a deploy.yml file (in config/ if you're doing a Rails project):
       repository: "https://github.com/username/project.git"
       branch: "staging"
       rake_env:
-        RAILS_ENV: 'production'	    
+        RAILS_ENV: 'production'
 
 then:
 
@@ -62,9 +62,9 @@ current local checkout.
   - You can do multi-project deployments, specifying deployment data in a single
     deploy.yml config file, or keep an entire directory of project deployment config files.
 
-  - You can separate per-deployment application configuration information (e.g., passwords, 
-    database configs, hoptoad/AWS/email config data, etc.) in separate repositories from 
-    the application, and whiskey\_disk will merge the correct data onto the deployed 
+  - You can separate per-deployment application configuration information (e.g., passwords,
+    database configs, hoptoad/AWS/email config data, etc.) in separate repositories from
+    the application, and whiskey\_disk will merge the correct data onto the deployed
     application at deployment time.  You can even share sets of configuration files among
     deployment targets that behave alike.
 
@@ -73,9 +73,9 @@ current local checkout.
     specify a config_branch and everyone can have their own local setup that just
     works.
 
-  - There's no before\_after\_before_after hooks.  You can use a well-defined rake hook and/or a 
+  - There's no before\_after\_before_after hooks.  You can use a well-defined rake hook and/or a
     bash script to run additional tasks.
- 
+
   - You can enable "staleness checks" so that deployments only happen if
     either the main repo, or the config repo (if you're using one) has
     changes that are newer than what is currently deployed.
@@ -85,10 +85,10 @@ current local checkout.
     deployment branch of choice!
 
   - You can deploy to multiple remote targets at once.  Currently this is limited
-    to one-after-the-other synchronous deployments, but we're thinking about 
+    to one-after-the-other synchronous deployments, but we're thinking about
     doing them in parallel once we're happy with the stability of this feature.
 
-  - Assign hosts to roles (e.g., "web", "db", "app") and vary the shell or rake 
+  - Assign hosts to roles (e.g., "web", "db", "app") and vary the shell or rake
     post-setup/post-deploy actions you run based on those roles.
 
   - Limit the actions you run after deployment based on whether files of interest
@@ -104,14 +104,14 @@ current local checkout.
 
 ### Dependencies ###
 
-On the server from which the whiskey_disk process will be kicked off:  
+On the server from which the whiskey_disk process will be kicked off:
 
  - ruby
  - rake
  - whiskey\_disk
- - ssh (if doing a remote deployment).  
+ - ssh (if doing a remote deployment).
 
-On the deployment target server (which may be the same as the first server):  
+On the deployment target server (which may be the same as the first server):
 
  - a bash-compatible shell
  - rsync (only if using a configuration repository)
@@ -136,7 +136,7 @@ As a rails plugin:
  - config files are YAML, with a section for each target.
 
 Known config file settings (if you're familiar with capistrano and vlad these should seem eerily familiar):
- 
+
     domain:              host or list of hosts on which to deploy (these are ssh connect strings)
                          can also optionally include role information for each host
     deploy_to:           path to which to deploy main application
@@ -153,7 +153,7 @@ Known config file settings (if you're familiar with capistrano and vlad these sh
 
 
 A simple config/deploy.yml might look like:
-  
+
     qa:
       domain: "ogc@qa.ogtastic.com"
       deploy_to: "/var/www/www.ogtastic.com"
@@ -217,6 +217,18 @@ You can even include a local deployment along with remote deployments, simply us
         RAILS_ENV: 'production'
 
 
+If you need special flags passed to ssh for a given domain, specify an array of `ssh_options` flags:
+
+    qa:
+      domain:
+      - name: "ogc@qa.ogtastic.com"
+        ssh_options:
+        - "-t"
+        - "-vv"
+        - "-p 443"
+      deploy_to: "/var/www/www.ogtastic.com"
+      repository: "git@ogtastic.com:www.ogtastic.com.git"
+
 ### Specifying domains, with or without roles ###
 
 There are a number of ways to specify domains (the ssh connection strings denoting the hosts
@@ -230,7 +242,7 @@ Just a single domain:
 Just a single domain, but specified as a one-item list:
 
     qa:
-      domain: 
+      domain:
       - "foo@qa.example.com"
 
 A list of multiple domains:
@@ -249,7 +261,7 @@ otherwise it's optional and superfluous):
       - name: "build@ci.example.com"
 
 It's also possible to assign various "roles" to the domains to which you deploy.  Some common usages would be
-"www", which might need a post\_deploy task which notifies some web server software (apache, nginx, passenger, 
+"www", which might need a post\_deploy task which notifies some web server software (apache, nginx, passenger,
 unicorn, etc.) that it should refresh the contents being served; or perhaps "db", which might need some set of
 post-deployment database migrations run (and which shouldn't be run from multiple servers).
 
@@ -273,7 +285,7 @@ While multiple roles for a domain must be specified as a list:
     production:
       domain:
       - name: "foo@appserver1.example.com"
-        roles: 
+        roles:
         - "web"
         - "app"
         - "db"
@@ -285,7 +297,7 @@ But domains with roles can be specified alongside simple domains as well:
       - name: "bar@demo.example.com"
       - "user@otherhost.domain.com"
       - name: "foo@appserver1.example.com"
-        roles: 
+        roles:
         - "web"
         - "app"
         - "db"
@@ -296,7 +308,7 @@ And, if you need to assign roles for a local deployment, you can do that as well
     local:
       domain:
       - name: "local"
-        roles: 
+        roles:
         - "web"
         - "app"
         - "db"
@@ -309,22 +321,22 @@ All that said, it's often simpler to refrain from carving up hosts into roles.
 ### post\_deploy\_script and post\_setup\_script ###
 
 Whiskey\_disk provides rake task hooks (deploy:post\_setup and deploy:post\_deploy) to allow running custom
-code after setup or deployment.  There are situations where it is desirable to run some commands prior to 
+code after setup or deployment.  There are situations where it is desirable to run some commands prior to
 running those rake tasks (e.g., if using bundler and needing to do a 'bundle install' before running rake).
 It may also be the case that the target system doesn't have rake (and/or ruby) installed, but some post-setup
 or post-deploy operations need to happen.  For these reasons, whiskey\_disk allows specifying a (bash-compatible)
 shell script to run after setup and/or deployment via the post\_deploy\_script and post\_setup\_script settings
-in the configuration file.  These scripts, when specified, are run immediately before running the deploy:post\_setup 
+in the configuration file.  These scripts, when specified, are run immediately before running the deploy:post\_setup
 or deploy:post\_deploy rake tasks, if they are present.
 
 The paths provided to post\_deploy\_script and post\_setup\_script can be either absolute or relative.  A path
-starting with a '/' is an absolute path, and the script specified should be at that exact location on the 
+starting with a '/' is an absolute path, and the script specified should be at that exact location on the
 target filesystem.  A path which does not start with a '/' is a relative path, and the script specified should
-be located at the specified path under the deployed application path.  This implies that it's possible to 
+be located at the specified path under the deployed application path.  This implies that it's possible to
 manage post\_setup and post\_deploy scripts out of a configuration repository.
 
 A config/deploy.yml using post\_deploy\_script and post\_setup\_script might look like this:
-  
+
     production:
       domain: "ogc@www.ogtastic.com"
       deploy_to: "/var/www/www.ogtastic.com"
@@ -335,7 +347,7 @@ A config/deploy.yml using post\_deploy\_script and post\_setup\_script might loo
       rake_env:
         RAILS_ENV: 'production'
 
-The post\_deploy\_script will be run from /var/www/www.ogtastic.com/bin/post-deploy.sh on the 
+The post\_deploy\_script will be run from /var/www/www.ogtastic.com/bin/post-deploy.sh on the
 target system.
 
 
@@ -356,10 +368,10 @@ role.  For example:
           Dir.chdir(RAILS_ROOT)
           system("mkdir -p log tmp")
         end
-      end        
+      end
 
       task :db_migrate_if_necessary do
-				Rake::Task['db:migrate'] if role? :db
+        Rake::Task['db:migrate'] if role? :db
       end
 
       # whytf is this even necessary?  Come on.  This should be built into ts:restart.
@@ -370,17 +382,17 @@ role.  For example:
           Rake::Task['ts:start'].invoke
         end
       end
-    
+
       task :bounce_passenger do
         if role? :www
           puts "restarting Passenger web server"
           Dir.chdir(RAILS_ROOT)
-          system("touch tmp/restart.txt")    
+          system("touch tmp/restart.txt")
         end
       end
 
       # etc...
-    
+
       task :post_setup  => [ :create_rails_directories ]
       task :post_deploy => [ :db_migrate_if_necessary, :thinking_sphinx_restart, :bounce_passenger ]
     end
@@ -399,36 +411,36 @@ are in an environment where the 'app' role is active but the 'web' role is not:
 
 
     $ wd_role web && rock || shocker
-         .-.     
-       .-.U|     
-       |U| | .-. 
-       | | |_|U| 
-       | | | | | 
+         .-.
+       .-.U|
+       |U| | .-.
+       | | |_|U|
+       | | | | |
       /|     ` |
-     | |       | 
-     |         | 
-      \        / 
-      |       |  
-      |       |  
-                 
+     | |       |
+     |         |
+      \        /
+      |       |
+      |       |
+
     $ wd_role app && rock || shocker
-       .-.       
-       |U|       
-       | |   .-. 
-       | |-._|U| 
-       | | | | | 
-      /|     ` | 
-     | |       | 
-     |         | 
-              / 
-      |       |  
-      |       |  
-    
+       .-.
+       |U|
+       | |   .-.
+       | |-._|U|
+       | | | | |
+      /|     ` |
+     | |       |
+     |         |
+              /
+      |       |
+      |       |
+
 
 
 
 ### Running whiskey\_disk from the command-line ###
-  
+
     % wd setup --to=<target>
     % wd setup --to=<project>:<target>
     % wd setup --to=foo:qa --path=/etc/whiskey_disk/deploy.yml
@@ -449,7 +461,7 @@ are in an environment where the 'app' role is active but the 'web' role is not:
 Also, `--debug` can be used to enable verbose output when running `wd setup` or `wd deploy`.
 
 Note that the wd command (unlike rake, which requires a Rakefile in the current directory) can be run from anywhere, so you can deploy any project, working from any path, and can even specify where to find the deployment YAML configuration file.
-  
+
 The --path argument can take either a file or a directory.  When given a file it will use that file as the configuration file.  When given a directory it will look in that directory for deploy/&lt;project&gt;/&lt;target&gt;.yml, then deploy/&lt;project&gt;.yml, then deploy/&lt;target&gt;.yml, then &lt;target&gt;.yml, and finally, deploy.yml.
 
 To make things even better, you can provide an URL as the --path argument and have a central location from which to pull deployment YAML data.  This means that you can centrally administer the definitive deployment information for the various projects and targets you manage.  This could be as simple as keeping them in a text file hosted on a web server, checking them into git and using github or gitweb to serve up the file contents on HEAD, or it could be a programmatically managed configuration management system returning dynamically-generated results.
@@ -480,7 +492,7 @@ The --only setting is used to tell a node what its name is, and to tell it not t
 
 ### A note about post\_{setup,deploy} Rake tasks
 
-If you want actions to run on the deployment target after you do a whiskey\_disk setup or whiskey\_disk deploy, 
+If you want actions to run on the deployment target after you do a whiskey\_disk setup or whiskey\_disk deploy,
 you will need to make sure that whiskey\_disk is available on the target system (either by gem installation,
 as a rails plugin in the Rails application to be deployed, or as a vendored library in the application to be
 deployed).  Whiskey\_disk provides the basic deploy:post\_setup and deploy:post\_deploy hooks which get called.
@@ -505,7 +517,7 @@ Do you want to run database migrations when no migrations have been added?  Why
       end
 
       task :compress_assets do
-        if changed?('public/stylesheets') or changed?('public/javascripts') 
+        if changed?('public/stylesheets') or changed?('public/javascripts')
           puts "Getting my asset munge on..."
           # do some expensive asset compression stuff
         end
@@ -578,7 +590,7 @@ environment variable to the string 'true' or 'yes':
 
 #### What's all this about a second repository for configuration stuff? ####
 
-This is completely optional, but we really are digging this, so maybe 
+This is completely optional, but we really are digging this, so maybe
 you should try it.  Basically it goes like this...
 
 We have a number of web applications that we manage.  Usually there's a
@@ -744,44 +756,46 @@ Notice that there are no separate trees for 'uat' and 'qa' targets.
 
  - Here is a sample of a lib/tasks/deploy.rake from a Rails application we deployed once upon a time:
 
-    RAILS_ENV=ENV['RAILS_ENV'] if ENV['RAILS_ENV'] and '' != ENV['RAILS_ENV']
-    Rake::Task['environment'].invoke
-    
-    require 'asset_cache_sweeper'
 
-    namespace :deploy do
-      task :create_rails_directories do
-        puts "creating log/ and tmp/ directories"
-        Dir.chdir(RAILS_ROOT)
-        system("mkdir -p log tmp")
-      end
-    
-      # note that the plpgsql language needs to be installed by the db admin at initial database creation :-/
-      task :setup_postgres_for_thinking_sphinx => [ :environment ] do
-        ThinkingSphinx::PostgreSQLAdapter.new(Product).setup
-      end
-    
-      # whytf is this even necessary?  Come on.  This should be built into ts:restart.
-      task :thinking_sphinx_restart => [:environment] do
-        Rake::Task['ts:stop'].invoke rescue nil
-        Rake::Task['ts:index'].invoke
-        Rake::Task['ts:start'].invoke
-      end
-    
-      task :bounce_passenger do
-        puts "restarting Passenger web server"
-        Dir.chdir(RAILS_ROOT)
-        system("touch tmp/restart.txt")    
-      end
-    
-      task :clear_asset_cache => [:environment] do
-        STDERR.puts "Expiring cached Assets for domains [#{AssetCacheSweeper.domains.join(", ")}]"
-        AssetCacheSweeper.expire
-      end
-    
-      task :post_setup => [ :create_rails_directories, :setup_postgres_for_thinking_sphinx ]
-      task :post_deploy => [ 'db:migrate', 'ts:config', :thinking_sphinx_restart, :bounce_passenger, :clear_asset_cache ]
-    end
+
+        RAILS_ENV=ENV['RAILS_ENV'] if ENV['RAILS_ENV'] and '' != ENV['RAILS_ENV']
+        Rake::Task['environment'].invoke
+
+        require 'asset_cache_sweeper'
+
+        namespace :deploy do
+          task :create_rails_directories do
+            puts "creating log/ and tmp/ directories"
+            Dir.chdir(RAILS_ROOT)
+            system("mkdir -p log tmp")
+          end
+
+          # note that the plpgsql language needs to be installed by the db admin at initial database creation :-/
+          task :setup_postgres_for_thinking_sphinx => [ :environment ] do
+            ThinkingSphinx::PostgreSQLAdapter.new(Product).setup
+          end
+
+          # whytf is this even necessary?  Come on.  This should be built into ts:restart.
+          task :thinking_sphinx_restart => [:environment] do
+            Rake::Task['ts:stop'].invoke rescue nil
+            Rake::Task['ts:index'].invoke
+            Rake::Task['ts:start'].invoke
+          end
+
+          task :bounce_passenger do
+            puts "restarting Passenger web server"
+            Dir.chdir(RAILS_ROOT)
+            system("touch tmp/restart.txt")
+          end
+
+          task :clear_asset_cache => [:environment] do
+            STDERR.puts "Expiring cached Assets for domains [#{AssetCacheSweeper.domains.join(", ")}]"
+            AssetCacheSweeper.expire
+          end
+
+          task :post_setup => [ :create_rails_directories, :setup_postgres_for_thinking_sphinx ]
+          task :post_deploy => [ 'db:migrate', 'ts:config', :thinking_sphinx_restart, :bounce_passenger, :clear_asset_cache ]
+        end
 
 ### Future Directions ###
 
@@ -806,10 +820,12 @@ The IRC channel is #whiskey_disk on freenode.
 ### Contributors ###
 
  - [Rick Bradley](https://github.com/rick): author
+ - [James Cox](https://github.com/imajes): bugfixes, prototyping, design help, code, feedback
  - [Rein Henrichs](https://github.com/reinh): design help, code
  - [Jeremy Holland](https://github.com/awebneck): code
  - [Kevin Barnes](https://github.com/vinbarnes): design help, code
  - [Alex Sharp](https://github.com/ajsharp): issues, real-world usage cases, design help
  - [Yossef Mendelssohn](https://github.com/ymendel): design help, proofreading
  - [Josh Moore](https://github.com/joshsmoore): bug-hunting
- - [Cristi Balan](https://github.com/evilchelu): feedback, proofreading
+
+ - Documentation/feedback: [Cristi Balan](https://github.com/evilchelu), [Hedgehog](http://github.com/hedgehog)