dbcp 0.0.1 → 0.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: dae551f41d7c976a088cafc8ebe340418b9e26ec
-  data.tar.gz: 1210fe47e8461e62afac77a58c238c346020e313
+  metadata.gz: e37c2566e5ac25d166c8fd27df6209e2f83863f1
+  data.tar.gz: 0960077bc0488bd979f3b97b751a940b3e6f11c4
 SHA512:
-  metadata.gz: 88e7b3c91a04ecf258f16cf848aff23a3925653b175cd35ccc0ff1fca586a6ed2b50a1325c463249c3468c0d1f2dc48a0d17638ce67e85742272cbd17dba82cd
-  data.tar.gz: d6adbc1c6773aaf9a6a1748e94c3683ba4f91772022f66172bba177df6d8135959a5406c9f420d7e595dfd79216fe145f95f11129cea6d610f0cb459edd28055
+  metadata.gz: 0d8b2f092e45700399fbfda1e91cdb0c80205ecf9670b07c5b5996d153ddf9570df48c57278a8427e7ec32350d11148356864f3a0c427d74dc462a45b0747ff0
+  data.tar.gz: fdde59cd6bd19e060aefd6a1c5999087ebeab62aa2c08c5f4135fadb320979bffb0e8b514256258fc65767646a990b16ec498bbd686aee5741af50b491d70bec

data/README.md

@@ -5,7 +5,9 @@
 
 Copy Postgres or MySQL databases between application environments.
 
-Setting an employee up to work on a web application for the first time is time consuming. Tools like [Vagrant](http://www.vagrantup.com) have made it easy to get your environment setup, but you still need to get your relational database setup. In rails you can load your `db/schema.rb` and hope that `db/seeds.rb` is well curated, but seldom has enough to let a developer hit the ground running. Working with your production database while developing is extremely convenient. The [parity](http://12factor.net/dev-prod-parity) helps preview database performance. It also makes investigating data-specific bugs much easier. The goal of `dbcp` is to make copying databases between development, staging, and production environments as easy as copying a file on your local disk.
+Setting an employee up to work on a web application for the first time is time consuming. Tools like [Vagrant](http://www.vagrantup.com) have made it easy to get your environment setup, but you still need to get your relational database setup. In rails you can load your `db/schema.rb` and hope that `db/seeds.rb` is well curated, but seldom has enough to let a developer hit the ground running. Working with your production database while developing is extremely convenient. The [parity](http://12factor.net/dev-prod-parity) helps preview database performance. It also makes investigating data-specific bugs much easier.
+
+The goal of `dbcp` is to make copying databases between development, staging, and production environments as easy as copying a file on your local disk. It's an adapter for platform-specific utilities like `pg_dump` and `mysqldump`, simplifies lookup of credentials using storage mechanisms you're already using, and handles transfer of dump files between hosts.
 
 ## A word of caution
 
@@ -37,30 +39,59 @@ Environment credentials can be defined in the following providers:
 
 Rails defines credentials for its database environments in a file at [`config/database.yml`](https://github.com/rails/rails/blob/master/guides/code/getting_started/config/database.yml). By default this file is generated with only development and test environments, but any additional environments added will be leveraged by `dbcp`. Although this is a rails convention, `dbcp` parses this file outside of any framework, so it will work even if you're using this convention in another framework.
 
-The database export or import can be executed on a remote host over ssh and then copied between environment hosts if you specify the remote host via an `ssh_uri` entry in the database.yml. This is helpful if the database host only allows connections from specific servers.
+The database export or import can be executed on a remote host over ssh and then copied between environment hosts if you specify the remote host via an `ssh_uri` entry in the database.yml. This is helpful if the database host only allows connections from specific servers. If your `ssh_uri` optionally includes a path to your application root on the remote server, dbcp will load the database credentials from the remote server's config/database.yml.
 
 Example config/database.yml:
 
 ```yaml
+# Local database
+development:
+  adapter:  postgresql
+  encoding: unicode
+  pool: 5
+  database: development_database
+  username: development_username
+  password: development_password
+
+# Remote database, credentials provided locally, executed from remote host over ssh
 staging:
   adapter:  postgresql
   database: staging_database
   username: staging_username
   password: staging_password
-  ssh_uri:  ssh://deploy@staging.example.com/www/staging.example.com/current
+  ssh_uri:  ssh://deploy@staging.example.com
+
+# Remote database, credentials fetched over ssh, executed from remote host over ssh
+production:
+  ssh_uri:  ssh://deploy@production.example.com/www/production.example.com/current
 ```
 
+    $ dbcp staging development
+
+### URI
+
+You can use a database URI in place of an environment name as follows:
+
+    $ dbcp postgres://my_username:my_pass@db.example.com/my_database development
+
 ## Roadmap
 
 The following features are pending:
 
 Providers:
 
-- URL passed in as environment
-- capistrano task
-- heroku, inferred from git remotes
+- Capistrano task
+- Heroku, environment name inferred from git remotes
 
 Features:
 
-- Reading configuration from a remote config/database.yml
 - Definable per-tool specific options, e.g. to allow pg_dump to provide a table exclusion list
+- URI Provider: specify an remote ssh execution host, perhaps using '@@' as a URI separator?
+
+Refactors:
+
+- Handle pg_restore warnings
+- Better logging
+- Better help
+
+[Open an issue](https://github.com/gabetax/dbcp/issues) if there's something else you'd like to see supported.

data/lib/dbcp/database.rb

@@ -1,5 +1,6 @@
 module Dbcp
   class Database
+    class BlankDatabaseDefinition < StandardError; end
     class UnsupportedDatabaseAdapter < StandardError; end
 
     class << self
@@ -13,6 +14,8 @@ module Dbcp
           MysqlDatabase
         when /postgres/
           PostgresDatabase
+        when nil, ''
+          raise BlankDatabaseDefinition.new("No database adapter was provided.")
         else
           raise UnsupportedDatabaseAdapter.new("Unsupported database adapter: #{adapter}")
         end
@@ -24,6 +27,7 @@ module Dbcp
       attribute :adapter
       attribute :database
       attribute :host, String, default: 'localhost'
+      attribute :port, Fixnum
       attribute :socket
       attribute :username
       attribute :password

data/lib/dbcp/databases/mysql_database.rb

@@ -1,21 +1,25 @@
 module Dbcp
   class MysqlDatabase < Database
     def export_command(snapshot_file)
-      %W[mysqldump #{socket_or_host} --user=#{username} --password=#{password} --add-drop-table --extended-insert --result-file=#{snapshot_file.path} #{database}].shelljoin
+      %W[mysqldump #{build_socket_or_host} #{build_port} --user=#{username} --password=#{password} --add-drop-table --extended-insert --result-file=#{snapshot_file.path} #{database}].reject(&:empty?).shelljoin
     end
 
     def import_command(snapshot_file)
-      %W[mysql #{socket_or_host} --user=#{username} --password=#{password} #{database}].shelljoin + ' < ' + snapshot_file.path.shellescape
+      %W[mysql #{build_socket_or_host} #{build_port} --user=#{username} --password=#{password} #{database}].reject(&:empty?).shelljoin + ' < ' + snapshot_file.path.shellescape
     end
 
     private
 
-    def socket_or_host
+    def build_socket_or_host
       if socket
         "--socket=#{socket}"
       else
         "--host=#{host}"
       end
     end
+
+    def build_port
+      "--port=#{port}" if host && port
+    end
   end
 end

data/lib/dbcp/databases/postgres_database.rb

@@ -1,11 +1,38 @@
 module Dbcp
   class PostgresDatabase < Database
     def export_command(snapshot_file)
-      %W[export PGPASSWORD=#{password}].shelljoin + '; ' + %W[pg_dump --host #{host} --username #{username} --file #{snapshot_file.path} --format c #{database}].shelljoin
+      build_password + [
+        'pg_dump',
+        '--host',     host,
+        build_port,
+        '--username', username,
+        '--file',     snapshot_file.path,
+        '--format',   'c',
+        database
+      ].flatten.compact.shelljoin
     end
 
     def import_command(snapshot_file)
-      %W[export PGPASSWORD=#{password}].shelljoin + '; ' + %W[pg_restore --host #{host} --username #{username} --clean --dbname #{database} #{snapshot_file.path}].shelljoin
+      build_password + [
+        'pg_restore',
+        '--host',     host,
+        build_port,
+        '--username', username,
+        '--dbname',   database,
+        '--clean',
+        snapshot_file.path
+      ].flatten.compact.shelljoin
     end
+
+    private
+
+    def build_password
+      %W[export PGPASSWORD=#{password}].shelljoin + '; '
+    end
+
+    def build_port
+      ['--port', port] if host && port
+    end
+
   end
 end

data/lib/dbcp/environment.rb

@@ -4,7 +4,8 @@ module Dbcp
 
   class Environment
     ENVIRONMENT_PROVIDERS = [
-      DatabaseYamlEnvironmentProvider.new('config/database.yml')
+      DatabaseYamlEnvironmentProvider.new('config/database.yml'),
+      UriEnvironmentProvider.new
     ]
 
     class << self

data/lib/dbcp/environment_providers/database_yaml_environment_provider.rb

@@ -27,11 +27,28 @@ module Dbcp
     end
 
     def build_environment(environment_name, environment_hash)
+      execution_host = ExecutionHost.build(environment_hash)
+
+      begin
+        database = Database.build(environment_hash)
+      rescue Database::BlankDatabaseDefinition => e
+        if execution_host.remote?
+          database = Database.build fetch_remote_environment_hash(execution_host)[environment_name]
+        else
+          raise e
+        end
+      end
+
       Environment.new({
         environment_name: environment_name,
-        database:         Database.build(environment_hash),
-        execution_host:   ExecutionHost.build(environment_hash)
+        database:         database,
+        execution_host:   execution_host
       })
     end
+
+    def fetch_remote_environment_hash(execution_host)
+      YAML.load execution_host.download "#{execution_host.path}/#{@database_yaml_path}"
+    end
+
   end
 end

data/lib/dbcp/environment_providers/uri_environment_provider.rb

@@ -0,0 +1,29 @@
+require 'uri'
+
+module Dbcp
+  class UriEnvironmentProvider
+    def find(environment_name)
+      uri = URI.parse(environment_name)
+      return nil unless uri.scheme
+
+      build_environment environment_name, uri
+    end
+
+    private
+
+    def build_environment(environment_name, uri)
+      Environment.new({
+        environment_name: environment_name,
+        database:         Database.build({
+          'adapter'  => uri.scheme,
+          'username' => uri.user,
+          'password' => uri.password,
+          'host'     => uri.host,
+          'port'     => uri.port,
+          'database' => uri.path[1..-1] # Trim leading '/'
+        }),
+        execution_host:   LocalExecutionHost.new
+      })
+    end
+  end
+end

data/lib/dbcp/execution_hosts/ssh_execution_host.rb

@@ -65,7 +65,8 @@ module Dbcp
       raise ExecutionError.new "Execution failed with exit code #{$?.exitstatus}. Command was: #{command}" if exitstatus > 0
     end
 
-    def download(source_path, destination_path)
+    # Omitting destination_path will return file contents as a string
+    def download(source_path, destination_path = nil)
       Net::SFTP.start host, username do |ssh|
         return ssh.download! source_path, destination_path
       end

data/lib/dbcp/version.rb

@@ -1,3 +1,3 @@
 module Dbcp
-  VERSION = "0.0.1"
+  VERSION = "0.1.0"
 end

data/lib/dbcp.rb

@@ -1,11 +1,11 @@
-require 'pry'
-
 require 'logger'
+require 'shellwords'
 require 'virtus'
 require 'dbcp/cli'
 require 'dbcp/database'
 require 'dbcp/databases/mysql_database'
 require 'dbcp/databases/postgres_database'
+require 'dbcp/environment_providers/uri_environment_provider'
 require 'dbcp/environment_providers/database_yaml_environment_provider'
 require 'dbcp/execution_host'
 require 'dbcp/execution_hosts/local_execution_host'

data/spec/fixtures/config/database.yml

@@ -3,6 +3,8 @@ development:
   encoding: unicode
   database: dev_database
   pool: 5
+  host: db.example.com
+  port: 6543
   username: dev_username
   password: dev_password
 
@@ -13,4 +15,7 @@ staging:
   pool: 5
   username: staging_username
   password: staging_password
+  ssh_uri: ssh://deploy@staging.example.com
+
+staging_ssh_only:
   ssh_uri: ssh://deploy@staging.example.com/www/staging.example.com/current

data/spec/fixtures/config/remote_database.yml

@@ -0,0 +1,7 @@
+staging_ssh_only:
+  adapter: postgresql
+  encoding: unicode
+  database: remote_staging_database
+  pool: 5
+  username: remote_staging_username
+  password: remote_staging_password

data/spec/lib/dbcp/cli_spec.rb

@@ -4,7 +4,8 @@ describe Dbcp::Cli do
   subject { Dbcp::Cli.new silent_stdout }
   let(:silent_stdout) { '/dev/null' }
 
-  before { Dir.chdir lib = File.expand_path('../../../fixtures', __FILE__) }
+  extend ExecuteInDirectory
+  execute_in_directory(File.expand_path('../../../fixtures', __FILE__))
 
   describe "#start" do
     context "success" do

data/spec/lib/dbcp/database_spec.rb

@@ -2,12 +2,16 @@ require 'spec_helper'
 
 describe Dbcp::Database do
   describe ".build" do
-    context "valid type" do
+    context "supported type" do
       specify { expect(Dbcp::Database.build 'adapter' => 'postgresql').to be_a(Dbcp::PostgresDatabase) }
     end
 
-    context "invalid type" do
-      specify { expect { Dbcp::Database.build 'adapter' => 'invalid'}.to raise_error(Dbcp::Database::UnsupportedDatabaseAdapter) }
+    context "no type" do
+      specify { expect { Dbcp::Database.build 'adapter' => nil}.to raise_error(Dbcp::Database::BlankDatabaseDefinition) }
+    end
+
+    context "unsupported type" do
+      specify { expect { Dbcp::Database.build 'adapter' => 'unsupported'}.to raise_error(Dbcp::Database::UnsupportedDatabaseAdapter) }
     end
 
   end

data/spec/lib/dbcp/environment_providers/database_yaml_environment_provider_spec.rb

@@ -2,7 +2,10 @@ require 'spec_helper'
 
 describe Dbcp::DatabaseYamlEnvironmentProvider do
   subject { Dbcp::DatabaseYamlEnvironmentProvider.new path }
-  let(:path) { File.expand_path('../../../../fixtures/config/database.yml', __FILE__) }
+  let(:path) { 'config/database.yml' }
+
+  extend ExecuteInDirectory
+  execute_in_directory(File.expand_path('../../../../fixtures', __FILE__))
 
   describe "#find" do
     context "when environment exists" do
@@ -12,6 +15,8 @@ describe Dbcp::DatabaseYamlEnvironmentProvider do
         expect(environment.database).to be_a Dbcp::Database
         expect(environment.environment_name).to  eq 'development'
         expect(environment.database.adapter).to  eq 'postgresql'
+        expect(environment.database.host).to     eq 'db.example.com'
+        expect(environment.database.port).to     eq 6543
         expect(environment.database.database).to eq 'dev_database'
         expect(environment.database.username).to eq 'dev_username'
         expect(environment.database.password).to eq 'dev_password'
@@ -30,10 +35,25 @@ describe Dbcp::DatabaseYamlEnvironmentProvider do
           expect(environment.execution_host).to be_a Dbcp::SshExecutionHost
         end
       end
+
+      context "without database definition, but with ssh_uri" do
+        let(:remote_yaml) { File.read remote_yaml_path }
+        let(:remote_yaml_path) { File.expand_path('../../../../fixtures/config/remote_database.yml', __FILE__) }
+        it "fetches database definition from database.yml on remote host" do
+          allow_any_instance_of(Dbcp::SshExecutionHost).to receive(:download).with('/www/staging.example.com/current/config/database.yml') { remote_yaml }
+          environment = subject.find 'staging_ssh_only'
+          expect(environment.database).to be_a Dbcp::PostgresDatabase
+          expect(environment.database.database).to eq 'remote_staging_database'
+          expect(environment.database.username).to eq 'remote_staging_username'
+          expect(environment.database.password).to eq 'remote_staging_password'
+        end
+      end
     end
+
     context "when environment doesn't exist" do
       specify { expect(subject.find 'does-not-exist').to be_nil }
     end
+
     context "when file doesn't exist" do
     let(:path) { File.expand_path('../../../../fixtures/config/database-does-not-exist.yml', __FILE__) }
       specify { expect(subject.find 'development').to be_nil }

data/spec/lib/dbcp/environment_providers/uri_environment_provider_spec.rb

@@ -0,0 +1,24 @@
+require 'spec_helper'
+
+describe Dbcp::UriEnvironmentProvider do
+  describe "#find" do
+    context "when a URI" do
+      let(:uri) { 'postgres://my_username:my_password@db.example.com:5432/my_database' }
+      it "returns an environment" do
+        environment = subject.find(uri)
+        expect(environment).to be_a Dbcp::Environment
+        expect(environment.environment_name).to eq uri
+        expect(environment.database).to be_a Dbcp::PostgresDatabase
+        expect(environment.database.username).to eq 'my_username'
+        expect(environment.database.password).to eq 'my_password'
+        expect(environment.database.host).to eq 'db.example.com'
+        expect(environment.database.database).to eq 'my_database'
+      end
+    end
+
+    context "when not a valid URI" do
+      let(:uri) { 'development'}
+      specify { expect(subject.find(uri)).to be_nil }
+    end
+  end
+end

data/spec/support/execute_in_directory.rb

@@ -0,0 +1,10 @@
+module ExecuteInDirectory
+  def execute_in_directory(path)
+    around(:each) do |example|
+      old = Dir.pwd
+      Dir.chdir path
+      example.run
+      Dir.chdir old
+    end
+  end
+end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: dbcp
 version: !ruby/object:Gem::Version
-  version: 0.0.1
+  version: 0.1.0
 platform: ruby
 authors:
 - Gabe Martin-Dempesy
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-04-07 00:00:00.000000000 Z
+date: 2014-04-09 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: virtus
@@ -146,23 +146,27 @@ files:
 - lib/dbcp/databases/postgres_database.rb
 - lib/dbcp/environment.rb
 - lib/dbcp/environment_providers/database_yaml_environment_provider.rb
+- lib/dbcp/environment_providers/uri_environment_provider.rb
 - lib/dbcp/execution_host.rb
 - lib/dbcp/execution_hosts/local_execution_host.rb
 - lib/dbcp/execution_hosts/ssh_execution_host.rb
 - lib/dbcp/version.rb
 - spec/fixtures/config/database.yml
+- spec/fixtures/config/remote_database.yml
 - spec/lib/dbcp/cli_spec.rb
 - spec/lib/dbcp/database_snapshot_file_spec.rb
 - spec/lib/dbcp/database_spec.rb
 - spec/lib/dbcp/databases/mysql_database_spec.rb
 - spec/lib/dbcp/databases/postgres_database_spec.rb
 - spec/lib/dbcp/environment_providers/database_yaml_environment_provider_spec.rb
+- spec/lib/dbcp/environment_providers/uri_environment_provider_spec.rb
 - spec/lib/dbcp/environment_spec.rb
 - spec/lib/dbcp/execution_host_spec.rb
 - spec/lib/dbcp/execution_hosts/local_execution_host_spec.rb
 - spec/lib/dbcp/execution_hosts/ssh_execution_host_spec.rb
 - spec/spec_helper.rb
 - spec/support/.keep
+- spec/support/execute_in_directory.rb
 homepage: https://github.com/gabetax/dbcp
 licenses:
 - MIT
@@ -189,15 +193,18 @@ specification_version: 4
 summary: ''
 test_files:
 - spec/fixtures/config/database.yml
+- spec/fixtures/config/remote_database.yml
 - spec/lib/dbcp/cli_spec.rb
 - spec/lib/dbcp/database_snapshot_file_spec.rb
 - spec/lib/dbcp/database_spec.rb
 - spec/lib/dbcp/databases/mysql_database_spec.rb
 - spec/lib/dbcp/databases/postgres_database_spec.rb
 - spec/lib/dbcp/environment_providers/database_yaml_environment_provider_spec.rb
+- spec/lib/dbcp/environment_providers/uri_environment_provider_spec.rb
 - spec/lib/dbcp/environment_spec.rb
 - spec/lib/dbcp/execution_host_spec.rb
 - spec/lib/dbcp/execution_hosts/local_execution_host_spec.rb
 - spec/lib/dbcp/execution_hosts/ssh_execution_host_spec.rb
 - spec/spec_helper.rb
 - spec/support/.keep
+- spec/support/execute_in_directory.rb