solutious-rudy 0.8.5 → 0.9.0

data/CHANGES.txt

@@ -1,18 +1,45 @@
 RUDY, CHANGES
 
-#### 0.9.0 (TBD) ###############################
 
-NOTE: The following changes are planned for the 0.9.0 release. 
-
-* group identifier will be changed to 'grp' from 'g'
-* machine identifier will be changes to 'mach' from 'm'
-* private keys are stored in ~/.ssh/
-* zone is now a single letter: a, b, c, etc... The long form (us-east-1b) will set the region too.
-* Disk syntax in the routines DSL will change. The DSL no longer accepts arguments, just the action (create, destroy, ...). 
-* Disk backups will be removed (temporarily -- probably until on of the early 1.x releases). 
-
-
-#### 0.8.5 (2009-06-??) ###############################
+#### 0.9.0-RC1 (2009-08-01) ###########################
+
+NOTE: This is a significant re-write from 0.8 (routines, metadata)
+
+* FIXED: Routine blocks now process directives in the order their given (in Ruby 1.8 and 1.9). 
+* FIXED: Region is now based on the zone. Corrects ambiguity when specifying zone and region.
+* FIXED: Reboot routines are now fully supported
+* FIXED: Unless position (-p) is specified, it's assumed that a command applies to the entire machine group. 
+* CHANGE: Rudy::Machine#inspect now prints in a standard Ruby inspect format
+* CHANGE: Routines have been completely rebuilt
+* CHANGE: Now requires Rye 0.8
+* CHANGE: before / after routine actions are now used for dependencies only.
+* CHANGE: Removed script_local, after_local, before_local from passthrough routines
+* CHANGE: Support before_local for startup routines
+* CHANGE: Support before_local and before_remote for reboot routines
+* CHANGE: Support after_local for shutdown routines
+* CHANGE: Removed machines config (now recommend uploading local files)
+* CHANGE: Group names now take the form: grp-us-east-1b-stage-app
+* CHANGE: private keys are now stored in ~/.ssh/
+* CHANGE: Removed create_image and prepare_image actions from rudy-ec2
+* CHANGE: local routine blocks now share a single Rye::Box instance
+* CHANGE: remote routine blocks now share a single Rye::Set instance
+* CHANGE: "privatekey" in accounts config is renamed to "pkey"
+* CHANGE: Reduced Rudy::AWS#execute_request timeout to 15 seconds.
+* CHANGE: Rudy:Disks, Rudy::Machines, and Rudy::Backups now use class methods
+* CHANGE: Rudy::AWS::EC2 now uses class methods
+* CHANGE: Renamed disk action 'snapshot' to 'archive'
+* ADDED: routines command for displaying available routines
+* ADDED: "rudy init" now authorizes public keys for user@localhost
+* ADDED: Parallel processing for remote machines
+* ADDED: Routines now have access to $global and $option (which are frozen)
+* ADDED: xremote and xlocal routine actions for quick skipping
+* ADDED: Dependency Attic 0.4
+* ADDED: New 'objects' and 'query' commands for rudy-sdb
+* ADDED: Execute routines on individual machines in a group
+* ADDED: rudy backups command
+
+
+#### 0.8.5 (2009-06-08) ###############################
 
 * FIXED: Bug in DiskHelper. mount method tried to attach when already attached. 
 * CHANGE: rudy config now displays account info (but keeps secret key hidden)
@@ -78,7 +105,7 @@ NOTE: The following changes are planned for the 0.9.0 release.
 * CHANGE: Moved non-rudy specific modules to separate projects (storable, sysinfo, annoy, 
           console -> Drydock, escape -> Rye, tryouts -> tryouts/).
 * CHANGE: Renamed restart routine to reboot to make it clear it's about the machine
-* CHANGE: Renamed Rudy::Disk to Rudy::MetaData::Disk
+* CHANGE: Renamed Rudy::Disk to Rudy::Metadata::Disk
 * CHANGE: Short alias for rudy-ec2 snapshot is now 's' (instead of 't')
 * CHANGE: Removed highline-1.5.1 from vendor (it's now available as a gem)
 * FIXED: Bug in Huxtable when loading global. Wasn't reloaded after config was updated. 
@@ -240,7 +267,21 @@ NOTE: This is a complete re-write from 0.1
 * ADDED: "rudy groups" overhaul. Display, creates, destroys groups. 
 
 
-
 #### 0.1 (2009-02-06) ###############################
 
 * Initial public release
+
+
+* TODO: Supply dns values in machines config
+* TODO: Rudy::Routines::Arcade. 
+  * install/uninstall: uses package management for the specific os impl
+  * compile: from source. 
+  * e.g. rudy -a experimental install git (for ruby-unix-osx)
+  * How to handle multiple package managers?
+  * How to handle multiple app versions? 
+* TODO: http://opensoul.org/2009/6/24/capistrano-git-and-ssh-keys
+* TODO: look for config file in ./.rudy, then ../.rudy, etc... (git-like)
+* TODO: FIX: When shutting down, warns about disks even when they don't exist (CLI::Routines)
+* TODO: Investigate SSH daemon on Windows
+* TODO: Allow colons in place of dashes (rudy publish:gem)
+* TODO: Supply machines config to routines blocks with only the specific machine group context 

data/README.rdoc

@@ -1,27 +1,19 @@
-= Rudy - v0.8 BETA
+= Rudy - v0.9-RC1
 
 <b>Not your grandparents' EC2 deployment tool.</b>
 
-Rudy is a command-line development and deployment tool for Amazon Web Services. It helps you build and manage machines in EC2 by organizing them into groups of _environments_ and _roles_. These are called _machine_ _groups_. <tt>stage</tt> is the default environment and <tt>app</tt> is the default role. You can run multiple machines with the same role. These are called _positions_. Rudy also supports running machine groups across availability _zones_. 
+Rudy is a development and deployment tool for Amazon Web Services. It helps you build and infrastructures in EC2 by organizing them into groups of _zones_,  _environments_, and _roles_. You can run multiple machines with the same role. When you put all this together, you have a unique name for every machine. The default machine is called <tt>m-us-east-1b-stage-app-01</tt> but you can also give them custom names. 
 
-When you put all this together, you have a unique name for every machine. The default machine is:
+All configuration is organized into the zones, environments, and roles that you specify (Rudy assumes positions are identical which is important for backups and scaling). And as you'd expect, the defaults can be changed too.
 
-             zone     env  role
-              v        v    v   
-        m-us-east-1b-stage-app-01
-        ^                       ^
-     "machine"                position
-
-All configuration is organized into the zones, environments, and roles that you specify (Rudy assumes positions are identical which is important for backups and scaling). And as you'd expect, the defaults can be changed too (in <tt>~/.rudy/config</tt>).
-
-<b>See Getting-Started[http://solutious.com/products/rudy/getting-started.html] for more info.</b>
+<b>See Getting-Started[http://solutious.com/projects/rudy/getting-started/] for more info.</b>
 
 
 == Project Status
 
 <em>This is a BETA release. That means Rudy's not ready for production use! See Project-Status[http://wiki.github.com/solutious/rudy/project-status].</em>
 
-I'm currently rebuilding[http://github.com/solutious/rudy/tree/metadata-refactor] the metadata implementation. This is the part of Rudy that stores your machine, disk, and backup infoz in SimpleDB so it's pretty important! The implementation in the 0.8.4 release (and this master branch) are functional but frail so I want to re-emphasize not to use Rudy for production right now.
+The 0.9 release is a major re-write and has many changes from previous releases. 0.9-RC1 is the first public release from the 0.9 branch and much of the code has not been tested. DO NOT USE IN PRODUCTION. Try it out have some fun and if you have the time send a patch or pull request!
 
 
 == Configuration
@@ -30,13 +22,13 @@ I'm currently rebuilding[http://github.com/solutious/rudy/tree/metadata-refactor
 
 The machines configuration describes the "physical" characteristics of your infrastructure. 
 
-    env :stage do         # Define an environment
+    env :stage do                   # Define an environment
       ami 'ami-e348af8a'         
                              
-      role :app do        # Define a role
-        addresses '11.22.33.44'  # Use elastic IPs
+      role :app do                  # Define a role
+        addresses '11.22.33.44'     # Use elastic IPs
                              
-        disks do          # Create EBS volumes
+        disks do                    # Define EBS volumes
           path "/rudy/disk1" do  
             size 100             
             device "/dev/sdr"     
@@ -49,18 +41,18 @@ The machines configuration describes the "physical" characteristics of your infr
 
 The routines configuration describes repeatable processes that you can execute on your machines. 
 
-    startup do            # $ rudy startup
+    startup do                      # $ rudy startup
       adduser :rudy       
-      authorize :rudy     # Enable passwordless login
+      authorize :rudy               # Enable passwordless login
                              
-      disks do                   # Create a disk 
-        restore "/rudy/disk1"    # from a backup
+      disks do                   
+        create "/rudy/disk1"        # Create a disk 
       end                       
                             
-      after :rudy do      # Run remote SSH commands
-        mkdir :p, "great"        # $ mkdir -p great
-        touch "great/scott"      # $ touch great/scott
-        ls :l, :a                # $ ls -l -a
+      remote :rudy do               # Run remote commands via SSH
+        mkdir :p, "great"           # $ mkdir -p great
+        touch "great/scott"         # $ touch great/scott
+        mysql_init :start 
       end                          
     end
 
@@ -68,15 +60,15 @@ The routines configuration describes repeatable processes that you can execute o
 
 == Features
 
-* Create complex environments with a single command
+* Create complex IT infrastructures from a simple configuration
   * Launch multiple instances 
   * Assign elastic IP addresses
   * Create EBS volumes, attach to instances, format, and mount
-  * Run scripts locally and remotely automatically
+* Run SSH commands on machine groups in parallel
 * Organize instances into environments and roles
 * Powerful command-line tools
   * <tt>$ rudy -u root ssh</tt>
-  * <tt>$ rudy -e testing -r database startup</tt>
+  * <tt>$ rudy -e testing -r database backup-mysql</tt>
 * Use _any Linux_ Amazon machine image (AMI) (partial Solaris support)
 * Complete command-line interface for EC2. See <tt>bin/rudy-ec2</tt>.
 
@@ -93,7 +85,7 @@ or via download:
 
 NOTE: <em>If you are not installing via RubyGems, you need to make sure the dependencies are in your LOAD_PATH (<tt>$:</tt>). Ryan Tomayko wrote a gist[http://gist.github.com/54177] about it.</em>
 
-<b>See Getting-Started[http://solutious.com/products/rudy/getting-started.html] for more info.</b>
+<b>See Getting-Started[http://solutious.com/projects/rudy/getting-started/] for more info.</b>
 
 
 == Pre-Requisites
@@ -102,8 +94,9 @@ NOTE: <em>If you are not installing via RubyGems, you need to make sure the depe
   * EC2[http://aws.amazon.com/ec2/]
   * SimpleDB[http://aws.amazon.com/simpledb/]
   * S3[http://aws.amazon.com/s3/]
+* OpenSSL[http://www.openssl.org/]
 * OpenSSH[http://www.openssh.com/]
-* Ruby[http://ruby-lang.org] 1.8.x, 1.9.x, or JRuby[http://jruby.codehaus.org] 1.2
+* Ruby[http://ruby-lang.org] 1.8.x, 1.9.x, or JRuby[http://jruby.codehaus.org] 1.3
   * Net::SSH[http://net-ssh.rubyforge.org/]
   * Net::SCP[http://net-ssh.rubyforge.org/]
   * amazon-ec2[http://github.com/grempe/amazon-ec2]
@@ -111,6 +104,7 @@ NOTE: <em>If you are not installing via RubyGems, you need to make sure the depe
   * Rye[http://github.com/delano/rye]
   * Drydock[http://github.com/delano/drydock]
   * Caesars[http://github.com/delano/caesars]
+  * Gibbler[http://github.com/delano/gibbler]
 
 
 == More Info
@@ -120,7 +114,7 @@ NOTE: <em>If you are not installing via RubyGems, you need to make sure the depe
 * Submit issues to the IssueTracker[http://github.com/solutious/rudy/issues]
 * Start a discussion on the GoogleGroup[http://groups.google.com/group/rudy-deployment]
 * Find some Inspiration[http://www.youtube.com/watch?v=CgaiIW5Rzes]
-* For all other inquires, email me directly: delano@solutious.com
+* For all other inquires, email me directly: delano (@solutious.com)
 
 
 == Thanks
@@ -142,10 +136,8 @@ NOTE: <em>If you are not installing via RubyGems, you need to make sure the depe
 
 == Contributions
 
-I'm very open to contributions! 
-
 * For bigger features please fork the git repo and send me a pull request. 
-* For small or single file changes, send me an email with the details: <tt>delano@solutious.com</tt>
+* For small or single file changes, send me an email with the details: Delano (@solutious.com)
 
 
 == Credits

data/Rudyfile

@@ -25,25 +25,26 @@
 #
 
 # ---------------------------------------------------------  MACHINES  --------
-# The machines block describes the "physical" characteristics of your machines. 
+# The machines block describes the 'physical' characteristics of your machines. 
 machines do
   
-  zone :"us-east-1b" do
+  region :'us-east-1' do
     ami 'ami-e348af8a'               # Alestic Debian 5.0, 32-bit (US)
   end
-  zone :"eu-west-1b" do
+  region :'eu-west-1' do
     ami 'ami-6ecde51a'               # Alestic Debian 5.0, 32-bit (EU)
   end
   
-  hostname :rudy                     # One of: :default, :rudy, "your-name"
+  hostname :rudy                     # One of: :default, :rudy, 'your-name'
   
-  # We've defined an environment called "stage" with one role: "app". 
+  # We've defined an environment called 'stage' with one role: 'app'. 
   # The configuration inside the env block is available to all its 
   # roles. The configuration inside the role blocks is available only
   # to machines in that specific role. 
   env :stage, :prod do
+    user :root                       # User to connect as
     size 'm1.small'                  # EC2 machine type for all machines
-                                     # in the "stage" environment
+                                     # in the 'stage' environment
     role :app do
       positions 1                    # Only 1 machine in stage-app
       #addresses '11.22.33.44'       # Define an elastic IP to reuse
@@ -63,20 +64,12 @@ machines do
     
     users do                         # Specify existing private keys per user
       rudy do                       
-        keypair "/path/2/private-key"
+        keypair '/path/2/private-key'
       end
     end
     
   end  
 
-  # The routines section below contains calls to local and remote
-  # scripts. The config contained in this block is made available
-  # those scripts in the form of a yaml file. The file is called
-  # rudy-config.yml. 
-  config do                 
-    dbmaster 'localhost'
-    newparam 573114
-  end
 end
 
 
@@ -94,7 +87,8 @@ end
 commands do
   allow :gem_install, '/usr/bin/gem', 'install', :V, '--no-rdoc', '--no-ri'
   allow :apt_get, 'apt-get', :y, :q
-  allow :floo, :time, 1 do
+  allow :rubycode do
+    puts "Some ruby code running in #{self}"
   end
 end
 
@@ -104,17 +98,18 @@ end
 routines do
   
   env :stage, :prod do               # We'll define routines for the stage-app
-    role :app do                     # and prod-app machine groups
+    role :app do                     # and prod-app machine groups.
+      
+      user :root                     # The default remote user
       
       startup do                     # $ rudy startup
         adduser :rudy                # Create a user called 'rudy'
-        authorize :rudy              # Enable passwordless logins as rudy
                                      #
         disks do                     # Define EBS volume routines
           create '/rudy/disk1'       # Create an EBS volume, attach it, give
         end                          # it a filesystem, and mount it.
                                      #
-        after :rudy do               # Run remote SSH commands after startup
+        remote :rudy do              # Run remote SSH commands after startup
           mkdir :p, 'great'          # $ mkdir -p great
           touch 'great/scott'        # $ touch great/scott
           ls :l, :a                  # $ ls -l -a *
@@ -122,7 +117,7 @@ routines do
       end                            
                                      
       shutdown do                    # $ rudy shutdown
-        before :root do              # Run remote SSH commands before shutdown
+        remote :root do              # Run remote SSH commands before shutdown
           uptime               
         end                               
         disks do                     
@@ -131,10 +126,10 @@ routines do
       end                            
       
       reboot do                      # $ rudy reboot
-        before :root do              # Run any pre-reboot tasks like stopping
+        before_remote do             # Run any pre-reboot tasks like stopping
           uptime                     # web servers and databases. 
         end                          #
-        after :root do               # Run any startup tasks like starting
+        remote do                    # Run any startup tasks like starting
           uptime                     # processes or initializing the filesystem
         end
       end
@@ -160,7 +155,7 @@ routines do
   # install essential libraries. 
   # See http://github.com/rudy/arcade
   sysupdate do                       # $ rudy sysupdate
-    before :root do                  
+    remote :root do                  
       apt_get 'update'               # Update debian / ubuntu 
       apt_get 'install', 'build-essential', 'sqlite3', 'libsqlite3-dev'
       apt_get 'install', 'apache2-prefork-dev', 'libapr1-dev'
@@ -170,16 +165,15 @@ routines do
   
   anything do                        # $ rudy anything
     before :uptime                   # Specify a dependency
-    script_local do                  # This is Ruby, so any valid syntax
+    local do                         # This is Ruby, so any valid syntax
       ls :l                          # can be used in the definitions. 
     end                              # See: SysInfo gem for more info.
   end
   
   uptime do                          # $ rudy uptime
-    script_local do
-      uptime
-    end
+    remote { uptime }                # Short block syntax
   end
+  
 end
 
 
@@ -188,12 +182,13 @@ end
 # non-boolean values are expected to be Symbols. 
 #
 defaults do
-  region :"us-east-1" 
-  zone :"us-east-1b"
+  zone :'us-east-1d'
   environment :stage
   role :app
-  user Rudy.sysinfo.user.to_sym      # This will be the user running Rudy
-  color false                        # Terminal colors? true/false
-  yes false                          # Auto-confirm? true/false
+  #user ENV['USER']                  # The default remote user
+  #localhost 'hostname'              # A local hostname instead of localhost
+  #color true                        # Terminal colors? true/false
+  #yes false                         # Auto-confirm? true/false
+  #keydir '~/.ssh/'                  # The path to store SSH keys
 end
 

data/bin/rudy

@@ -9,7 +9,7 @@
 # 
 
 $:.unshift File.join(File.dirname(__FILE__), '..', 'lib') # Put our local lib in first place
-#$:.unshift File.join(File.dirname(__FILE__), '..', '..', 'drydock', 'lib') 
+$:.unshift File.join(File.dirname(__FILE__), '..', '..', 'drydock', 'lib') 
 
 #$SAFE = 1   # require is unsafe in Ruby 1.9??
 
@@ -20,43 +20,65 @@ require 'rudy/cli'
 # Command-line interface for /bin/rudy
 class RudyCLI < Rudy::CLI::Base
   
-  debug :off
   default :machines       # when no command is provided
   trawler :passthrough    # unknown command names will forward here.
   
   # ------------------------------------------  RUDY GLOBALS  --------
   # ------------------------------------------------------------------
   
-  global :e, :environment, String, "Connect to the specified environment (ie: #{Rudy::DEFAULT_ENVIRONMENT})"
-  global :r, :role, String, "Connect to a machine with the specified role (ie: #{Rudy::DEFAULT_ROLE})"
-  global :p, :position, String, "Position in the machine in its group (ie: #{Rudy::DEFAULT_POSITION})"
-
-
+  global :e, :environment, String, "Connect to the specified environment (e.g. #{Rudy::DEFAULT_ENVIRONMENT})"
+  global :r, :role, String, "Connect to a machine with the specified role (e.g. #{Rudy::DEFAULT_ROLE})"
+  global :p, :position, String, "Position in the machine in its group"
+  global :b, :bucket, String, "An S3 bucket name (used when creating images)"
+  global :t, :testrun, "Test run. Don't execute action (EXPERIMENTAL)."
+  global :P, :parallel, "Execute remote commands in parallel (EXPERIMENTAL)."
+  global :F, :force, "Force an action despite warnings"
+  
   # ------------------------------------------  RUDY OBJECTS  --------
   # ------------------------------------------------------------------
   
+  about "View Routines"
+  usage "rudy routines"
+  usage "rudy routines -l"
+  option :l, :all, "Display routines for all environments and roles"
+  command :routines => Rudy::CLI::Routines
+  command_alias :routines, :r
+  
   about "View Machines"
+  usage "rudy"
+  usage "rudy machines"
+  usage "rudy machines -l"
   option :l, :all, "Display machines for all environments and roles"
   action :W, :wash, "Wash machine metadata."
   command :machines => Rudy::CLI::Machines
   command_alias :machines, :m
   
   about "View Disks"
+  usage "rudy disks"
+  usage "rudy disks -l"
   action :W, :wash, "Wash disk metadata."
   option :b, :backups, "Display backups"
   option :l, :all, "Display all disks"
   command :disks => Rudy::CLI::Disks
   command_alias :disks, :d
-  
+
+  about "View Backups"
+  usage "rudy backups"
+  usage "rudy backups -l"
+  action :W, :wash, "Wash backups metadata."
+  option :l, :all, "Display all backups"
+  command :backups => Rudy::CLI::Backups
+  command_alias :backups, :b
+
   about "View raw metadata"
   usage "rudy metadata"
-  usage "rudy metadata -l -o [disk|back|m]"
+  usage "rudy metadata -l -r [disk|back|m]"
   usage "rudy metadata -D object-id"
-  option :o, :otype, String, "Object type. One of: disk, back, m"
+  option :r, :rtype, String, "Record type. One of: disk, back, m (default)"
   option :l, :all, "Display metadata for all environments and roles"
   action :D, :delete, "Delete an object"
   argv :oid
-  command :metadata => Rudy::CLI::MetaData
+  command :metadata => Rudy::CLI::Metadata
   
   about "Log in to a machine"
   command :ssh => Rudy::CLI::Machines
@@ -119,20 +141,25 @@ class RudyCLI < Rudy::CLI::Base
     
     Rudy::Huxtable.update_config
     
-    unless File.exists?(Rudy::CONFIG_FILE)
+    unless File.exists? Rudy::CONFIG_FILE
       Rudy::Config.init_config_dir
     end
   
     begin
 
-      unless Rudy::Huxtable.domain_exists?
+      if Rudy::Huxtable.domain_exists?
+        puts "SimpleDB domain #{Rudy::Huxtable.domain} already exists"
+      else
         puts "Creating SimpleDB domain #{Rudy::Huxtable.domain}"
         Rudy::Huxtable.create_domain
-        puts "Initialized"
-      else
-        puts "Already Initialized"
       end
-    
+      
+      user, host = Rudy.sysinfo.user, Rudy::Huxtable.global.localhost
+
+      puts "Authorizing public keys for #{user}@#{host}"
+      rbox = Rye::Box.new host
+      rbox.authorize_keys_local
+      
     rescue Rudy::AWS::SDB::NoSecretKey, 
            Rudy::AWS::SDB::NoAccessKey, 
            Rudy::NoConfig => ex
@@ -208,13 +235,8 @@ class RudyCLI < Rudy::CLI::Base
   end
 end
 
-# We call Drydock specifically otherwise it will run at_exit. Rye also
-# uses at_exit for shutting down the ssh-agent. Ruby executes at_exit
-# blocks in reverse order so if Drydock is required first, it's block
-# will run after Rye shuts down the ssh-agent. 
 begin
   Drydock.run!(ARGV, STDIN) if Drydock.run? && !Drydock.has_run?
-
 rescue Drydock::ArgError, Drydock::OptError => ex
   STDERR.puts ex.message
   STDERR.puts ex.usage
@@ -223,12 +245,10 @@ rescue Drydock::InvalidArgument => ex
 rescue Rudy::Error => ex
   STDERR.puts ex.message
   STDERR.puts ex.backtrace if Drydock.debug?
-rescue => ex
-  STDERR.puts "ERROR (#{ex.class.to_s}): #{ex.message}"
-  STDERR.puts ex.backtrace if Drydock.debug?
 rescue Interrupt
   puts "#{$/}Exiting... "
   exit 1
-rescue SystemExit
-  # Don't balk
+rescue => ex
+  STDERR.puts "ERROR (#{ex.class.to_s}): #{ex.message}"
+  STDERR.puts ex.backtrace if Drydock.debug?
 end