cap-taffy 0.0.2

data/History.txt

@@ -0,0 +1,9 @@
+== 0.0.2 / 2009-09-14
+* 1 minor enhancement
+  * Added authorizing SSH public keys on remote server(s).
+
+== 0.0.1 / 2009-09-14
+
+* 1 major enhancement
+  * Initial release.
+  * Added db push/pull.

data/README.md

@@ -0,0 +1,83 @@
+Capistrano Taffy (Database, SSH recipes)
+================================================
+
+**Capistrano recipes for deploying databases and other common tasks (managing database.yml, importing/exporting/transfering databases, SSH authorization etc.)**
+
+Features
+------------------------------------------------
+
+* Adds database transfer recipes (via [`Taps`]("http://github.com/ricardochimal/taps"))
+* Authorize SSH access
+* Manage `database.yml` (Soon.)
+
+[`Taps`]("http://github.com/ricardochimal/taps") is great, but having to SSH into my deployment to run the `Taps` server, as well as 
+figure out the proper local/remote database urls, is a pain. I knew the [`Heroku`]("http://github.com/heroku/heroku") gem
+had it figured out; I present the Capistrano friendly version.
+
+If you are new to `Taps`, check out this [introduction to `Taps`]("http://adam.blog.heroku.com/past/2009/2/11/taps_for_easy_database_transfers/") by [Adam Wiggins]("http://github.com/adamwiggins").
+
+Installation
+------------------------------------------------
+
+    gem install cap-taffy
+
+Database Transfer
+------------------------------------------------
+
+> _Dependency:_ The [`Taps`]("http://github.com/ricardochimal/taps") gem is required on any server(s) you'll be transferring databases to (`:app` role) including your development machine (where you'll be running `cap` tasks from). Run:
+
+>     gem install taps
+
+> _Dependency:_ The [`Heroku`]("http://github.com/heroku/heroku") gem is required on your development machine.
+
+>     gem install heroku
+
+### Usage
+
+To start, add the following to your `Capfile`
+
+    require 'cap-taffy/db'
+
+`Taffy` will start a `Taps` server on port 5000 by default, so make sure port 5000 (or w/e port you end up using) is open on your `:app` server(s)
+
+Then you can use:
+
+    cap db:push         # Or to specify a different port: cap db:push -s taps_port=4321
+    cap db:pull         # Or to specify a different port: cap db:pull -s taps_port=4321
+    
+
+#### SSH Local Forwarding
+
+> Some of you may not be able/want to open up ports on your servers. Instead, you can run:
+
+>     ssh -N -L[port]:127.0.0.1:[port] [user]@[remote-server]
+> And then run:
+
+>     cap db:push -s taps_port=[port] -s local=true
+> Substituing the appropriate values for [port], [user], and [remote-server].
+> #### Sample Usage
+> >     ssh -N -L4321:127.0.0.1:4321 henry@load-test
+> >     cap db:push -s taps_port=4321 -s local=true
+
+SSH Access
+------------------------------------------------
+
+#### Usage
+
+Add the following to your `Capfile`
+
+    require 'cap-taffy/ssh'
+
+Using a public key generated from `ssh-keygen` (e.g. `ssh-keygen -t rsa`), to authorize access:
+
+    cap ssh:authorize       # authorizes local public key for SSH access to remote server(s)
+
+
+Managing `database.yml`
+------------------------------------------------
+
+> Much needed and coming soon.
+
+Credits
+------------------------------------------------
+Thanks to developers of the [`Taps`]("http://adam.blog.heroku.com/past/2009/2/11/taps_for_easy_database_transfers/") gem and the [`Heroku`]("http://github.com/heroku/heroku") gem. And of course, Capistrano, awesome!

data/Rakefile

@@ -0,0 +1,34 @@
+# Look in the tasks/setup.rb file for the various options that can be
+# configured in this Rakefile. The .rake files in the tasks directory
+# are where the options are used.
+
+begin
+  require 'bones'
+  Bones.setup
+rescue LoadError
+  begin
+    load 'tasks/setup.rb'
+  rescue LoadError
+    raise RuntimeError, '### please install the "bones" gem ###'
+  end
+end
+
+ensure_in_path 'lib'
+require 'cap-taffy'
+
+task :default => 'spec:run'
+
+PROJ.name = 'cap-taffy'
+PROJ.authors = 'Henry Hsu'
+PROJ.email = 'henry@qlane.com'
+PROJ.url = 'http://by.qlane.com'
+PROJ.version = CapTaffy::VERSION
+PROJ.rubyforge.name = 'cap-taffy'
+PROJ.readme_file = "README.md"
+PROJ.gem.dependencies = ['heroku', 'taps', 'capistrano']
+PROJ.gem.development_dependencies << ["mocha"]
+PROJ.description = "Capistrano recipes for deploying databases and other common tasks."
+PROJ.summary = "Capistrano recipes for deploying databases (managing database.yml, importing/exporting/transfering databases, etc.)"
+PROJ.ignore_file = '.gitignore'
+PROJ.spec.opts << '--color'
+PROJ.exclude << "bin"

data/cap-taffy.gemspec

@@ -0,0 +1,45 @@
+# -*- encoding: utf-8 -*-
+
+Gem::Specification.new do |s|
+  s.name = %q{cap-taffy}
+  s.version = "0.0.2"
+
+  s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
+  s.authors = ["Henry Hsu"]
+  s.date = %q{2009-09-17}
+  s.description = %q{Capistrano recipes for deploying databases and other common tasks.}
+  s.email = %q{henry@qlane.com}
+  s.extra_rdoc_files = ["History.txt"]
+  s.files = ["History.txt", "README.md", "Rakefile", "cap-taffy.gemspec", "lib/cap-taffy.rb", "lib/cap-taffy/db.rb", "lib/cap-taffy/parse.rb", "lib/cap-taffy/ssh.rb", "spec/cap-taffy/db_spec.rb", "spec/cap-taffy/parse_spec.rb", "spec/cap-taffy/ssh_spec.rb", "spec/cap-taffy_spec.rb", "spec/spec.opts", "spec/spec_helper.rb"]
+  s.homepage = %q{http://by.qlane.com}
+  s.rdoc_options = ["--main", "README.md"]
+  s.require_paths = ["lib"]
+  s.rubyforge_project = %q{cap-taffy}
+  s.rubygems_version = %q{1.3.5}
+  s.summary = %q{Capistrano recipes for deploying databases (managing database.yml, importing/exporting/transfering databases, etc.)}
+
+  if s.respond_to? :specification_version then
+    current_version = Gem::Specification::CURRENT_SPECIFICATION_VERSION
+    s.specification_version = 3
+
+    if Gem::Version.new(Gem::RubyGemsVersion) >= Gem::Version.new('1.2.0') then
+      s.add_runtime_dependency(%q<heroku>, [">= 0"])
+      s.add_runtime_dependency(%q<taps>, [">= 0"])
+      s.add_runtime_dependency(%q<capistrano>, [">= 0"])
+      s.add_development_dependency(%q<bones>, [">= 2.5.1"])
+      s.add_development_dependency(%q<mocha>, [">= 0"])
+    else
+      s.add_dependency(%q<heroku>, [">= 0"])
+      s.add_dependency(%q<taps>, [">= 0"])
+      s.add_dependency(%q<capistrano>, [">= 0"])
+      s.add_dependency(%q<bones>, [">= 2.5.1"])
+      s.add_dependency(%q<mocha>, [">= 0"])
+    end
+  else
+    s.add_dependency(%q<heroku>, [">= 0"])
+    s.add_dependency(%q<taps>, [">= 0"])
+    s.add_dependency(%q<capistrano>, [">= 0"])
+    s.add_dependency(%q<bones>, [">= 2.5.1"])
+    s.add_dependency(%q<mocha>, [">= 0"])
+  end
+end

data/lib/cap-taffy/db.rb

@@ -0,0 +1,239 @@
+require 'rubygems'
+begin
+  gem 'taps', '>= 0.2.8', '< 0.3.0'
+  require 'taps/client_session'
+rescue LoadError
+  error "Install the Taps gem to use db commands. On most systems this will be:\nsudo gem install taps"
+end
+
+require File.join(File.dirname(__FILE__), %w[.. cap-taffy]) unless defined?(CapTaffy)
+require File.join(File.dirname(__FILE__), 'parse')
+require 'digest/sha1'
+
+module CapTaffy::Db
+  extend self
+
+  # Detects the local database url for +env+.
+  #
+  # Looks for <tt>config/database.yml</tt>.
+  def local_database_url(env)
+    return "" unless File.exists?(Dir.pwd + '/config/database.yml')
+    db_config = YAML.load(File.read(Dir.pwd + '/config/database.yml'))
+
+    CapTaffy::Parse.database_url(db_config, env)
+  end
+
+  # Detects the remote database url for +env+ and the current Capistrano +instance+.
+  #
+  # Looks for <tt>config/database.yml</tt> in the +current_path+.
+  def remote_database_url(instance, env)
+    db_yml = instance.capture "cat #{instance.current_path}/config/database.yml"
+    db_config = YAML::load(db_yml)
+
+    CapTaffy::Parse.database_url(db_config, env)
+  end
+
+  # The default server port the Taps server is started on.
+  def default_server_port
+    5000
+  end
+
+  # Generates the remote url used by Taps push/pull.
+  #
+  # ==== Parameters
+  # 
+  # * <tt>:login, :password, :host, :port</tt> - See #run.
+  #
+  # ==== Examples
+  #
+  #   login = fetch(:user)
+  #   password = tmp_pass(login)                                               # returns asdkf239udjhdaks (for example)
+  #   remote_url(:login => login, :password => password, :host => 'load-test') # returns http://henry:asdkf239udjhdaks@load-test:5000
+  def remote_url(options={})
+    host = options[:host]
+    port = options[:port] || default_server_port
+    host += ":#{port}"
+    url = CapTaffy::Parse.new.uri_hash_to_url('username' => options[:login], 'password' => options[:password], 'host' => host, 'scheme' => 'http', 'path' => '')
+
+    url.sub(/\/$/, '')
+  end
+
+  # Generates a temporary password to be used for the Taps server command.
+  def tmp_pass(user)
+    Digest::SHA1.hexdigest("--#{Time.now.to_s}--#{user}--")
+  end
+
+  # A quick start for a Taps client.
+  #
+  # <tt>local_database_url</tt> and <tt>remote_url</tt> refer to the options for the Taps gem (see #run).
+  def taps_client(local_database_url, remote_url, &blk) # :yields: client
+    Taps::Config.chunksize = 1000
+    Taps::Config.database_url = local_database_url
+    Taps::Config.remote_url = remote_url
+    Taps::Config.verify_database_url
+
+    Taps::ClientSession.quickstart do |client|
+      yield client
+    end
+  end
+
+  # Generates the server command used to start a Taps server
+  #
+  # ==== Parameters
+  # * <tt>:remote_database_url, :login, :password</tt> - See #run.
+  # * <tt>:port</tt> - The +port+ the Taps server is on. If given and different from #default_server_port, appends <tt>--port=[port]</tt> to command.
+  def server_command(options={})
+    remote_database_url, login, password, port = options[:remote_database_url], options[:login], options[:password], options[:port]
+    port_argument = ''
+    port_argument = " --port=#{port}" if port && port != default_server_port
+
+    "taps server #{remote_database_url} #{login} #{password}#{port_argument}"
+  end
+
+  # The meat of the operation. Runs operations after setting up the Taps server.
+  # 
+  # 1. Runs the <tt>taps</tt> taps command to start the Taps server (assuming Sinatra is running on Thin)
+  # 2. Wait until the server is ready 
+  # 3. Execute block on Taps client
+  # 4. Close the connection(s) and bid farewell.
+  #
+  # ==== Parameters
+  # * <tt>:remote_database_url</tt> - Refers to local database url in the options for the Taps server command (see Taps Options).
+  # * <tt>:login</tt> - The login for +host+. Usually what's in <tt>set :user, "the user"</tt> in <tt>deploy.rb</tt>
+  # * <tt>:password</tt> - The temporary password for the Taps server.
+  # * <tt>:port</tt> - The +port+ the Taps server is on. If not given, defaults to #default_server_port.
+  # * <tt>:local_database_url</tt> - Refers to the local database url in the options for Taps client commands (see Taps Options).
+  #
+  # ==== Taps Options
+  #
+  # <tt>taps</tt>
+  #   server <local_database_url> <login> <password> [--port=N]        Start a taps database import/export server
+  #   pull <local_database_url> <remote_url> [--chunksize=N]           Pull a database from a taps server
+  #   push <local_database_url> <remote_url> [--chunksize=N]           Push a database to a taps server
+  #
+  # ==== Examples
+  #
+  #   task :push do
+  #     login = fetch(:user)
+  #     password = Time.now.to_s
+  #     CapTaffy.Db.run(self, { :login => login, :password => password, :remote_database_url => "sqlite://test_production", :local_database_url => "sqlite://test_development" }) do |client|
+  #       client.cmd_send
+  #     end
+  #   end
+  def run(instance, options = {} , &blk) # :yields: client
+    options[:port] ||= default_server_port
+    remote_database_url, login, password, port, local_database_url = options[:remote_database_url], options[:login], options[:password], options[:port], options[:local_database_url]
+    force_local = options.delete(:local)
+
+    data_so_far = ""
+    instance.run CapTaffy::Db.server_command(options) do |channel, stream, data|
+      data_so_far << data
+      if data_so_far.include? ">> Listening on 0.0.0.0:#{port}, CTRL+C to stop"
+        host = force_local ? '127.0.0.1' : channel[:host]
+        remote_url = CapTaffy::Db.remote_url(options.merge(:host => host))
+
+        CapTaffy::Db.taps_client(local_database_url, remote_url) do |client|
+          yield client
+        end
+
+        data_so_far = ""
+        channel.close
+        channel[:status] = 0
+      end
+    end
+  end
+
+  class InvalidURL < RuntimeError # :nodoc:
+  end
+end
+
+Capistrano::Configuration.instance.load do
+  namespace :db do
+    # Executes given block.
+    # If this is a dry run, any raised exceptions will be caught and +returning+ is returned.
+    # If this is not a dry run, any exceptions will be raised as expected.
+    def dry_run_safe(returning = nil, &block) # :yields:
+      begin
+        yield
+      rescue Exception => e
+        raise e unless dry_run
+        return returning
+      end
+    end
+
+    task :detect, :roles => :app do
+      @remote_database_url = dry_run_safe('') { CapTaffy::Db.remote_database_url(self, 'production') }
+      @local_database_url = dry_run_safe('') { CapTaffy::Db.local_database_url('development') }
+    end
+
+    desc <<-DESC
+      Push a local database into the app's remote database.
+      
+      Performs push from local development database to remote production database.
+      Opens a Taps server on port 5000. (Ensure port is opened on the remote server).
+      
+        # alternately, specify a different port
+        cap db:push -s taps_port=4321
+        
+      For the security conscious:
+      
+        # use ssh local forwarding (ensure [port] is available on both endpoints)
+        ssh -N -L[port]:127.0.0.1:[port] [user]@[remote-server]
+        
+        # then push locally
+        cap db:push -s taps_port=[port] -s local=true
+    DESC
+    task :push, :roles => :app do      
+      detect
+
+      login = fetch(:user)
+      password = CapTaffy::Db.tmp_pass(login)
+
+      logger = Capistrano::Logger.new
+      logger.important "Auto-detected remote database: #{@remote_database_url}" if @remote_database_url != ''
+      logger.important "Auto-detected local database: #{@local_database_url}" if @local_database_url != ''
+
+      options = {:remote_database_url => @remote_database_url, :login => login, :password => password, :local_database_url => @local_database_url, :port => variables[:taps_port]}
+      options.merge!(:local => true) if variables[:local]
+
+      CapTaffy::Db.run(self, options) do |client|
+        client.cmd_send
+      end
+    end
+
+    desc <<-DESC
+      Pull the app's database into a local database.
+      
+      Performs pull from remote production database to local development database.
+      Opens a Taps server on port 5000. (Ensure port is opened on the remote server).
+        
+        # alternately, specify a different port
+        cap db:pull -s taps_port=4321
+        
+      For the security conscious:
+
+        # use ssh local forwarding (ensure [port] is available on both endpoints)
+        ssh -N -L[port]:127.0.0.1:[port] [user]@[remote-server]
+
+        # then pull locally
+        cap db:pull -s taps_port=[port] -s local=true
+    DESC
+    task :pull, :roles => :app do
+      detect
+
+      login = fetch(:user)
+      password = CapTaffy::Db.tmp_pass(login)
+
+      logger = Capistrano::Logger.new
+      logger.important "Auto-detected remote database: #{@remote_database_url}" if @remote_database_url != ''
+      logger.important "Auto-detected local database: #{@local_database_url}" if @local_database_url != ''
+
+      options = {:remote_database_url => @remote_database_url, :login => login, :password => password, :local_database_url => @local_database_url, :port => variables[:taps_port]}
+      options.merge!(:local => true) if variables[:local]
+
+      CapTaffy::Db.run(self, options) do |client|
+        client.cmd_receive
+      end
+    end
+  end
+end

data/lib/cap-taffy/parse.rb

@@ -0,0 +1,53 @@
+require 'rubygems'
+begin
+  require 'heroku'
+  require 'heroku/commands/base'
+  require 'heroku/commands/db'
+rescue LoadError
+  error "Install the Heroku gem. On most systems this will be:\nsudo gem install taps"
+end
+
+module CapTaffy
+  class Parse < Heroku::Command::Db
+    class << self
+      attr_accessor :instance
+
+      # Modified from :parse_database_yml in heroku/command/db.rb
+      #
+      # Accepts a complete +db_config+ hash and +env+ and parses for a database_url accordingly.
+      def database_url(db_config, env)
+        raise Invalid, "please pass me a valid Hash loaded from a database YAML file" unless db_config
+        conf = db_config[env]
+        raise Invalid, "missing '#{env}' database in #{db_config.inspect}" unless conf
+
+        self.instance ||= CapTaffy::Parse.new
+
+        case conf['adapter']
+        when 'sqlite3'
+          return "sqlite://#{conf['database']}"
+        when 'postgresql'
+          uri_hash = self.instance.conf_to_uri_hash(conf)
+          uri_hash['scheme'] = 'postgres'
+          return self.instance.uri_hash_to_url(uri_hash)
+        else
+          return self.instance.uri_hash_to_url(self.instance.conf_to_uri_hash(conf))
+        end
+      end
+    end
+
+    # Override to do nothing on #new
+    def initialize # :nodoc:
+
+    end
+
+    # Override to pass-through on #escape
+    def escape(string)
+      string
+    end
+
+    public :uri_hash_to_url, :conf_to_uri_hash
+
+    class Invalid < RuntimeError # :nodoc:
+    end
+  end
+end

data/lib/cap-taffy/ssh.rb

@@ -0,0 +1,18 @@
+require File.join(File.dirname(__FILE__), %w[.. cap-taffy]) unless defined?(CapTaffy)
+
+module CapTaffy::SSH
+
+end
+
+Capistrano::Configuration.instance.load do
+  namespace :ssh do
+    desc <<-DESC
+      Authorize SSH access for local computer on remote computers(s).
+    DESC
+    task :authorize do
+      public_key = File.read(File.expand_path(File.join(%w[~/ .ssh id_rsa.pub]))).chop
+
+      run %Q[if [ ! -f ~/.ssh/authorized_keys ]; then mkdir -p ~/.ssh && touch ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys; fi && if [ -z "$(grep "^#{public_key}$" ~/.ssh/authorized_keys)" ]; then echo "#{public_key}" >> ~/.ssh/authorized_keys && echo "Public key on '$CAPISTRANO:HOST$' authorized at '#{Time.now.to_s}'"; else echo "Public key on '$CAPISTRANO:HOST$' is already authorized."; fi]
+    end
+  end
+end

data/lib/cap-taffy.rb

@@ -0,0 +1,45 @@
+
+module CapTaffy
+
+  # :stopdoc:
+  VERSION = '0.0.2'
+  LIBPATH = ::File.expand_path(::File.dirname(__FILE__)) + ::File::SEPARATOR
+  PATH = ::File.dirname(LIBPATH) + ::File::SEPARATOR
+  # :startdoc:
+
+  # Returns the version string for the library.
+  #
+  def self.version
+    VERSION
+  end
+
+  # Returns the library path for the module. If any arguments are given,
+  # they will be joined to the end of the libray path using
+  # <tt>File.join</tt>.
+  #
+  def self.libpath( *args )
+    args.empty? ? LIBPATH : ::File.join(LIBPATH, args.flatten)
+  end
+
+  # Returns the lpath for the module. If any arguments are given,
+  # they will be joined to the end of the path using
+  # <tt>File.join</tt>.
+  #
+  def self.path( *args )
+    args.empty? ? PATH : ::File.join(PATH, args.flatten)
+  end
+
+  # Utility method used to require all files ending in .rb that lie in the
+  # directory below this file that has the same name as the filename passed
+  # in. Optionally, a specific _directory_ name can be passed in such that
+  # the _filename_ does not have to be equivalent to the directory.
+  #
+  def self.require_all_libs_relative_to( fname, dir = nil )
+    dir ||= ::File.basename(fname, '.*')
+    search_me = ::File.expand_path(
+        ::File.join(::File.dirname(fname), dir, '**', '*.rb'))
+
+    Dir.glob(search_me).sort.each {|rb| require rb}
+  end
+
+end