capistrano-db_sync 0.0.2

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: 560f9902b7556b03a2206f1b3125d68d906000ed
4
+ data.tar.gz: fc422d26be8a12e80bae35f88e9918286439a3ec
5
+ SHA512:
6
+ metadata.gz: 001d12a9c6bbfce2689c49d47d6c9c8752d7792fa28355149b120306da264a33e40f4047ca41255067fb57ec4080832611006699619bed491dd009abbc2d2ca8
7
+ data.tar.gz: 1b0d4fdfd2210b57bedba454ef62446c952196462e5affd89effece893ba10d6fcf3b531d704b573ecfe837b72b1b60bc07ec0111c0ba3aaa3bccda5f9141909
data/.gitignore ADDED
@@ -0,0 +1,34 @@
1
+ *.gem
2
+ *.rbc
3
+ /.config
4
+ /coverage/
5
+ /InstalledFiles
6
+ /pkg/
7
+ /spec/reports/
8
+ /test/tmp/
9
+ /test/version_tmp/
10
+ /tmp/
11
+
12
+ ## Specific to RubyMotion:
13
+ .dat*
14
+ .repl_history
15
+ build/
16
+
17
+ ## Documentation cache and generated files:
18
+ /.yardoc/
19
+ /_yardoc/
20
+ /doc/
21
+ /rdoc/
22
+
23
+ ## Environment normalisation:
24
+ /.bundle/
25
+ /lib/bundler/man/
26
+
27
+ # for a library or gem, you might want to ignore these files since the code is
28
+ # intended to run in multiple environments; otherwise, check them in:
29
+ Gemfile.lock
30
+ .ruby-version
31
+ .ruby-gemset
32
+
33
+ # unless supporting rvm < 1.11.0 or doing something fancy, ignore this:
34
+ .rvmrc
data/Gemfile ADDED
@@ -0,0 +1,15 @@
1
+ source 'https://rubygems.org'
2
+
3
+ # Specify your gem's dependencies in capistrano-pg_sync.gemspec
4
+ gemspec
5
+
6
+ group :development, :test do
7
+ gem "bundler", "~> 1.7"
8
+ gem "rake", "~> 10.0"
9
+ gem "pry-byebug"
10
+ end
11
+
12
+ group :test do
13
+ gem "minitest"
14
+ gem "mocha"
15
+ end
data/LICENSE ADDED
@@ -0,0 +1,22 @@
1
+ The MIT License (MIT)
2
+
3
+ Copyright (c) 2015 Rafael Sales
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
22
+
data/README.md ADDED
@@ -0,0 +1,83 @@
1
+ ## Capistrano::DBSync
2
+
3
+ Fast and sophisticated remote database import using the best of **Postgres 9.2.x**
4
+
5
+ ### Features
6
+
7
+ * Allows dump data selectively - partial table data or no data
8
+ * Uses Postgres parallel restore
9
+ * Uses Postgres custom dump format that is automatically compressed
10
+
11
+ ### Requirements
12
+
13
+ * Capistrano 3.x
14
+ * Postgres 9.2.x
15
+ * It was tested with Rails only, but it is expected to work in any project containing
16
+ `config/database.yml` file in both local and remote machine.
17
+
18
+ ### Installation
19
+
20
+ 1. Add this line to your application's Gemfile:
21
+
22
+ ```ruby
23
+ gem 'capistrano-pg_sync', require: false
24
+ ```
25
+
26
+ 2. Define your custom settings, if needed. We suggest to locate this file at `lib/capistrano/tasks/db_sync.rake`.
27
+ Capistrano 3.x should load all `*.rake` files by default in `Capfile`.
28
+ [See the complete configuration reference](/lib/capistrano/db_sync/configuration.rb)
29
+
30
+ ```ruby
31
+ require 'capistrano/db_sync'
32
+
33
+ set :db_sync_options, -> do
34
+ {
35
+ # Hash mapping a table name to a query with data selection or nil in case no data
36
+ # is wanted for a table. Tables not listed here will be dumped entirely.
37
+ data_selection: {
38
+ posts: "SELECT * FROM posts WHERE created_at > NOW() - interval '60 days'",
39
+ comments: "SELECT * FROM comments WHERE created_at > NOW() - interval '30 days'",
40
+ likes: nil
41
+ },
42
+
43
+ local: {
44
+ cleanup: false, # If the downloaded dump directory should be removed after restored
45
+ pg_jobs: 2, # Number of jobs to run in parallel on pg_restore
46
+ },
47
+
48
+ remote: {
49
+ cleanup: true, # If the remote dump directory should be removed after downloaded
50
+ }
51
+ }
52
+ end
53
+ ```
54
+
55
+ ### Usage
56
+
57
+ ```sh-session
58
+ $ cap production db_sync:import
59
+ ```
60
+
61
+ ### How it works
62
+
63
+ The following steps describe what happens when executing `cap production db_sync:import`:
64
+
65
+ 1. SSH into production server with primary db role on capistrano stages configuration
66
+ 2. Connect to the remote Postgres using credentials of `config/database.yml` in the
67
+ deployed server
68
+ 3. Dump the database schema, data, triggers, constraints, rules and indexes
69
+ 4. Download the compressed dump files to local machine
70
+ 5. Restore the dumps in local machine in following sequence
71
+ 1. database schema
72
+ 2. data for tables with entire data dumped
73
+ 3. data for tables with partial data specified in configuration
74
+ 4. triggers, constraints, rules and indexes
75
+
76
+
77
+ ### Contributing
78
+
79
+ 1. Fork it ( https://github.com/rafaelsales/capistrano-db_sync/fork )
80
+ 2. Create your feature branch (`git checkout -b my-new-feature`)
81
+ 3. Commit your changes (`git commit -am 'Add some feature'`)
82
+ 4. Push to the branch (`git push origin my-new-feature`)
83
+ 5. Create a new Pull Request
data/Rakefile ADDED
@@ -0,0 +1,7 @@
1
+ require "bundler/gem_tasks"
2
+ require 'rake/testtask'
3
+
4
+ Rake::TestTask.new do |t|
5
+ t.libs += ["spec", "lib"]
6
+ t.test_files = FileList['spec/**/*_spec.rb']
7
+ end
@@ -0,0 +1,22 @@
1
+ # coding: utf-8
2
+ lib = File.expand_path('../lib', __FILE__)
3
+ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
4
+ require 'capistrano/db_sync/version'
5
+
6
+ Gem::Specification.new do |spec|
7
+ spec.name = "capistrano-db_sync"
8
+ spec.version = Capistrano::DBSync::VERSION
9
+ spec.authors = ["Rafael Sales"]
10
+ spec.email = ["rafaelcds@gmail.com"]
11
+ spec.summary = %q{A capistrano task to import remote Postgres databases}
12
+ spec.description = %q{Fast download and restore dumps using edge features of Postgres 9.x}
13
+ spec.homepage = "https://github.com/rafaelsales/capistrano-db_sync"
14
+ spec.license = "MIT"
15
+
16
+ spec.files = `git ls-files -z`.split("\x0")
17
+ spec.executables = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
18
+ spec.test_files = spec.files.grep(%r{^(test|spec|features)/})
19
+ spec.require_paths = ["lib"]
20
+
21
+ spec.add_runtime_dependency "capistrano", ">= 3.0.0"
22
+ end
@@ -0,0 +1,61 @@
1
+ module Capistrano::DBSync
2
+ class Configuration
3
+ extend Forwardable
4
+
5
+ DEFAULT_OPTIONS = ->(cap) do
6
+ {
7
+ # Hash mapping a table name to a query or nil in case no data is wanted for a table.
8
+ # E.g.: {
9
+ # posts: "SELECT * FROM posts WHERE created_at > NOW() - interval '60 days'",
10
+ # comments: "SELECT * FROM comments WHERE created_at > NOW() - interval '30 days'",
11
+ # likes: nil
12
+ # }
13
+ data_selection: {},
14
+
15
+ data_sync_confirmation: true, # Ask for user input confirmation
16
+
17
+ local: {
18
+ cleanup: false, # If the downloaded dump directory should be removed after restored
19
+
20
+ pg_jobs: 1, # Number of jobs to run in parallel on pg_restore
21
+
22
+ working_dir: "./tmp",
23
+ env: ENV.fetch('RAILS_ENV', 'development'),
24
+ },
25
+
26
+ remote: {
27
+ cleanup: true, # If the remote dump directory should be removed after downloaded
28
+
29
+ working_dir: "/tmp",
30
+ env: cap.fetch(:stage).to_s,
31
+ },
32
+ }
33
+ end
34
+
35
+ def initialize(cap_instance = Capistrano.env)
36
+ @cap = cap_instance
37
+ @options = load_options
38
+ end
39
+
40
+ def load_options
41
+ user_options = cap.fetch(:db_sync_options)
42
+ DEFAULT_OPTIONS.call(cap).deep_merge(user_options)
43
+ end
44
+
45
+ def data_sync_confirmed?
46
+ skip = options[:data_sync_confirmation].to_s.downcase == "false"
47
+ skip || prompt("Confirm replace local database with remote database?")
48
+ end
49
+
50
+ def_delegators :@options, :[], :fetch
51
+
52
+ private
53
+
54
+ attr_reader :cap, :options
55
+
56
+ def prompt(message, prompt = "(y)es, (n)o")
57
+ cap.ask(:prompt_answer, "#{message} #{prompt}")
58
+ (cap.fetch(:prompt_answer) =~ /^y|yes$/i) == 0
59
+ end
60
+ end
61
+ end
@@ -0,0 +1,35 @@
1
+ module Capistrano::DBSync
2
+ module Executor
3
+ class Base
4
+
5
+ # +side+ must be :local or :remote
6
+ def initialize(cap, config, side)
7
+ @cap = cap
8
+ @config = config
9
+ @session_id = Time.now.strftime("%Y-%m-%d-%H%M%S")
10
+ @side = side
11
+ end
12
+
13
+ def working_dir
14
+ File.join config[side][:working_dir]
15
+ end
16
+
17
+ def env
18
+ config[side][:env].to_s
19
+ end
20
+
21
+ def cleanup?
22
+ config[side][:cleanup]
23
+ end
24
+
25
+ private
26
+
27
+ def load_db_config!(config_file_contents)
28
+ yaml = YAML.load(ERB.new(config_file_contents).result)
29
+ @db_config = yaml[env].tap { |db_config| Postgres.validate!(db_config) }
30
+ end
31
+
32
+ attr_reader :cap, :config, :db_config, :side, :session_id
33
+ end
34
+ end
35
+ end
@@ -0,0 +1,38 @@
1
+ require "fileutils"
2
+
3
+ module Capistrano::DBSync
4
+ module Executor
5
+ class Local < Base
6
+ def initialize(cap, config)
7
+ super(cap, config, :local)
8
+ load_db_config!(File.read File.join("config", "database.yml"))
9
+ end
10
+
11
+ def restore!(dump_dir)
12
+ importer(dump_dir).restore(jobs: config[:local][:pg_jobs]).each do |cmd|
13
+ cap.info "Running locally: #{cmd}"
14
+ system(cmd)
15
+ end
16
+
17
+ cap.info "Completed database restore."
18
+ ensure
19
+ clean_dump_if_needed!(dump_dir)
20
+ end
21
+
22
+ private
23
+
24
+ def clean_dump_if_needed!(dump_dir)
25
+ if cleanup?
26
+ FileUtils.rm_rf dump_dir
27
+ cap.info "Removed #{dump_dir} locally."
28
+ else
29
+ cap.info "Leaving #{dump_dir} locally. Use \"local: { cleanup: true }\" to remove."
30
+ end
31
+ end
32
+
33
+ def importer(dump_dir)
34
+ Postgres::Importer.new(dump_dir, db_config)
35
+ end
36
+ end
37
+ end
38
+ end
@@ -0,0 +1,56 @@
1
+ module Capistrano::DBSync
2
+ module Executor
3
+ class Remote < Base
4
+ def initialize(cap, config)
5
+ super(cap, config, :remote)
6
+ load_db_config! cap.capture("cat #{File.join cap.current_path, 'config', 'database.yml'}")
7
+ end
8
+
9
+ # Returns the dump directory location that was downloaded to local
10
+ # machine, which is based on +local_working_dir+.
11
+ def dump_and_download_to!(local_working_dir)
12
+ dump!
13
+ download_to!(local_working_dir)
14
+ ensure
15
+ clean_dump_if_needed!
16
+ end
17
+
18
+ private
19
+
20
+ def dump!
21
+ cap.execute "mkdir -p #{dump_dir}"
22
+
23
+ exporter.dump(data_selection: config[:data_selection]).each do |cmd|
24
+ cap.execute cmd
25
+ end
26
+ end
27
+
28
+ def download_to!(local_working_dir)
29
+ system "mkdir -p #{local_working_dir}"
30
+ cap.download! dump_dir, local_working_dir, recursive: true
31
+
32
+ cap.info "Completed database dump and download."
33
+ File.join(local_working_dir, File.basename(dump_dir))
34
+ end
35
+
36
+ def clean_dump_if_needed!
37
+ if cleanup?
38
+ cap.execute "rm -rf #{dump_dir}"
39
+ cap.info "Removed #{dump_dir} from the server."
40
+ else
41
+ cap.info "Leaving #{dump_dir} on the server. Use \"remote: { cleanup: true}\" to remove."
42
+ end
43
+ end
44
+
45
+ private
46
+
47
+ def exporter
48
+ Postgres::Exporter.new(dump_dir, db_config)
49
+ end
50
+
51
+ def dump_dir
52
+ File.join(working_dir, "dump_#{session_id}_#{db_config['database']}")
53
+ end
54
+ end
55
+ end
56
+ end
@@ -0,0 +1,71 @@
1
+ class Capistrano::DBSync::Postgres::CLI
2
+ def initialize(config)
3
+ @config = config
4
+ end
5
+
6
+ def dump(to_file, db, options = [])
7
+ "#{with_pw} pg_dump #{credentials} #{format_args} -f #{to_file} #{options.join(' ')} #{db}"
8
+ end
9
+
10
+ def restore(from_file, db, options = [])
11
+ "#{with_pw} pg_restore #{credentials} #{format_args} -d #{db} #{options.join(' ')} #{from_file}"
12
+ end
13
+
14
+ def drop_db(db)
15
+ psql %Q|DROP DATABASE IF EXISTS "#{db}";|
16
+ end
17
+
18
+ def create_db(db)
19
+ psql %Q|CREATE DATABASE "#{db}";|
20
+ end
21
+
22
+ def rename_db(old_db, new_db)
23
+ psql %Q|ALTER DATABASE "#{old_db}" RENAME TO "#{new_db}";|
24
+ end
25
+
26
+ def clone_db(new_db, template_db)
27
+ psql %Q|CREATE DATABASE "#{new_db}" WITH TEMPLATE "#{template_db}";|
28
+ end
29
+
30
+ def psql(command, db = "postgres")
31
+ normalized_command = command.gsub('"', '\"').gsub(/\s\s+|\n/, " ")
32
+ %Q|#{with_pw} psql #{credentials} -d #{db} -c "#{normalized_command}"|
33
+ end
34
+
35
+ def kill_processes_for_db(db)
36
+ psql <<-SQL.gsub("$", "\\$")
37
+ SELECT pg_terminate_backend(pg_stat_activity.pid)
38
+ FROM pg_stat_activity
39
+ WHERE pg_stat_activity.datname = $$#{db}$$
40
+ AND pid <> pg_backend_pid();
41
+ SQL
42
+ end
43
+
44
+ def copy_to_file(to_compressed_file, db, query)
45
+ psql "\\COPY (#{query}) TO PROGRAM 'gzip > #{to_compressed_file}'", db
46
+ end
47
+
48
+ def copy_from_file(from_compressed_file, db, table)
49
+ psql "\\COPY #{table} FROM PROGRAM 'gunzip --to-stdout #{from_compressed_file}'", db
50
+ end
51
+
52
+ private
53
+
54
+ def format_args
55
+ "--no-acl --no-owner --format=custom"
56
+ end
57
+
58
+ def credentials
59
+ credentials_params = []
60
+ credentials_params << "-U #{config['username']}" unless config.fetch('username', '').empty?
61
+ credentials_params << "-h #{config['host']}" unless config.fetch('host', '').empty?
62
+ credentials_params << "-p #{config['port']}" unless config.fetch('port', '').empty?
63
+ credentials_params.join(" ")
64
+ end
65
+
66
+ def with_pw
67
+ "PGPASSWORD='#{config['password']}'" unless config.fetch('password', '').empty?
68
+ end
69
+
70
+ attr_reader :config, :session_id
71
+ end
@@ -0,0 +1,61 @@
1
+ module Capistrano::DBSync
2
+ class Postgres::Exporter
3
+
4
+ # +working_dir+: The location where the dump files will be stored for dump or read for restore.
5
+ # +config+: database configuration hash with following skeleton:
6
+ # {
7
+ # "database" => "faceburger_production",
8
+ # "username" => "fb_prod",
9
+ # "password" => "BestBurger",
10
+ # "host" => "10.20.30.40",
11
+ # "port" => "5432"
12
+ # }
13
+ def initialize(working_dir, config)
14
+ @working_dir = working_dir
15
+ @config = config
16
+ @cli = Postgres::CLI.new(config)
17
+ end
18
+
19
+ # Returns a set of commands to dump a database with table data selection support.
20
+ #
21
+ # +db+ (optional): Database name to dump
22
+ # +data_selection+ (optional): A hash mapping a table name to a query or nil in
23
+ # case no data is wanted for a table.
24
+ #
25
+ # Example:
26
+ #
27
+ # dump("/tmp/dump",
28
+ # data_selection:
29
+ # posts: "SELECT * FROM posts WHERE created_at > NOW() - interval '60 days'",
30
+ # comments: "SELECT * FROM comments WHERE created_at > NOW() - interval '30 days'",
31
+ # likes: nil
32
+ # },
33
+ # db: "faceburger_production")
34
+ #
35
+ # This outputs commands that will generate dump files as:
36
+ # /tmp/dump/0001-faceburger_production.schema -- will contain db schema and data except
37
+ # for tables posts, comments and likes
38
+ # /tmp/dump/0002-posts.table -- will contain partial data of table posts
39
+ # /tmp/dump/0003.comments.table -- will contain partial data of table comments
40
+ #
41
+ def dump(db = config["database"], data_selection: {})
42
+ file_namer = Postgres::FileNameGenerator.new(working_dir)
43
+ exclude_tables_args = data_selection.keys.map { |table| %Q|--exclude-table-data="#{table}"| }
44
+
45
+ [
46
+ cli.dump(file_namer.next(db, :schema), db, [exclude_tables_args]),
47
+ *dump_partial_selected_data(db, file_namer, data_selection)
48
+ ]
49
+ end
50
+
51
+ private
52
+
53
+ def dump_partial_selected_data(db, file_namer, data_selection)
54
+ data_selection.map do |table, query|
55
+ cli.copy_to_file(file_namer.next(table, :table), db, query) if query
56
+ end.compact
57
+ end
58
+
59
+ attr_reader :working_dir, :config, :cli
60
+ end
61
+ end
@@ -0,0 +1,22 @@
1
+ class Capistrano::DBSync::Postgres::FileNameGenerator
2
+ def initialize(path)
3
+ @path = path
4
+ @sequence = 0
5
+ end
6
+
7
+ # Generates sequential file names.
8
+ # Examples:
9
+ # next("faceburger", :schema) => 0001-faceburger.schema
10
+ # next("posts", :table) => 0002-posts.table
11
+ # next("comments", :table) => 0003-comments.table
12
+ def next(name, extension)
13
+ raise ArgumentError unless [:schema, :table].include?(extension)
14
+ @sequence += 1
15
+ File.join(@path, "%04i-#{name}.#{extension}" % @sequence)
16
+ end
17
+
18
+ def self.extract_name(file_path)
19
+ file_name = File.basename(file_path)
20
+ file_name.scan(/\d+-(.*)\.(schema|table)$/).flatten.first
21
+ end
22
+ end
@@ -0,0 +1,79 @@
1
+ module Capistrano::DBSync
2
+ class Postgres::Importer
3
+
4
+ # +working_dir+: The location where the dump files will be stored for dump or read for restore.
5
+ # +config+: database configuration hash with following skeleton:
6
+ # {
7
+ # "database" => "faceburger_production",
8
+ # "username" => "fb_prod",
9
+ # "password" => "BestBurger",
10
+ # "host" => "10.20.30.40",
11
+ # "port" => "5432"
12
+ # }
13
+ def initialize(working_dir, config)
14
+ @working_dir = working_dir
15
+ @config = config
16
+ @cli = Postgres::CLI.new(config)
17
+ end
18
+
19
+ # Returns a set of commands to dump a database with table data selection support.
20
+ #
21
+ # +db+ (optional): Database name to restore data into.
22
+ # +jobs+ (optional): Number of concurrent jobs that Postgres will run to restore.
23
+ #
24
+ # The +working_dir+ should contain files with following name and extension patterns:
25
+ #
26
+ # /tmp/dump/0001-faceburger_production.schema -- contains db schema and data except
27
+ # for tables posts and comments
28
+ # /tmp/dump/0002-posts.table -- contains partial data of table posts
29
+ # /tmp/dump/0003.comments.table -- contains partial data of table comments
30
+ def restore(db = config["database"], jobs: 1)
31
+ temp_db = "#{db}_#{Time.now.strftime("%Y%m%d%H%M%S")}"
32
+
33
+ [
34
+ cli.kill_processes_for_db(temp_db),
35
+ cli.drop_db(temp_db),
36
+ cli.create_db(temp_db),
37
+ *restore_files_to(temp_db, jobs),
38
+ cli.kill_processes_for_db(db),
39
+ cli.drop_db(db),
40
+ cli.rename_db(temp_db, db)
41
+ ]
42
+ end
43
+
44
+ private
45
+
46
+ # Commands to restore pg_dump output file and partial tables data files generated by
47
+ # Postgres COPY command.
48
+ def restore_files_to(to_db, jobs)
49
+ [
50
+ restore_pg_dump_file(to_db, %w(pre-data data), jobs),
51
+ *restore_table_copy_files(to_db),
52
+ restore_pg_dump_file(to_db, %w(post-data), jobs)
53
+ ]
54
+ end
55
+
56
+ # The +sections+ argument must match one of the following supported sections on pg_dump command:
57
+ # - Pre-data: include all other data definition items.
58
+ # - Data: contains actual table data, large-object contents, and sequence values.
59
+ # - Post-data: include definitions of indexes, triggers, rules, and constraints other
60
+ # than validated check constraints.
61
+ def restore_pg_dump_file(to_db, sections = %w(pre-data data post-data), jobs)
62
+ files = Dir.glob(File.join(working_dir, "*.schema"))
63
+ raise "Expected to have only one pg_dump file, but found these: #{files.inspect}" if files.size > 1
64
+
65
+ sections_args = sections.map { |section| "--section=#{section}" }
66
+ cli.restore(files.first, to_db, [*sections_args, "--jobs=#{jobs}"])
67
+ end
68
+
69
+ def restore_table_copy_files(to_db)
70
+ files = Dir.glob(File.join(working_dir, "*.table"))
71
+ files.map do |file|
72
+ table_name = Postgres::FileNameGenerator.extract_name(file)
73
+ cli.copy_from_file(file, to_db, table_name)
74
+ end
75
+ end
76
+
77
+ attr_reader :working_dir, :config, :cli
78
+ end
79
+ end
@@ -0,0 +1,9 @@
1
+ module Capistrano::DBSync
2
+ module Postgres
3
+ def self.validate!(config)
4
+ unless %w(postgresql pg).include? config['adapter']
5
+ raise NotImplementedError, "Database adapter #{config['adapter']} is not supported"
6
+ end
7
+ end
8
+ end
9
+ end
@@ -0,0 +1,25 @@
1
+ require "capistrano"
2
+ require "capistrano/version"
3
+ Bundler.require(:development)
4
+
5
+ namespace :db_sync do
6
+ desc <<-DESC
7
+ Synchronize your local database using remote database data.
8
+ Usage: $ cap <stage> db:pull
9
+ DESC
10
+
11
+ task :import do
12
+ config = Capistrano::DBSync::Configuration.new(self)
13
+
14
+ if config.data_sync_confirmed?
15
+ on roles(:db, primary: true) do
16
+ local = Capistrano::DBSync::Executor::Local.new(self, config)
17
+ remote = Capistrano::DBSync::Executor::Remote.new(self, config)
18
+
19
+ downloaded_dir = remote.dump_and_download_to! config[:local][:working_dir]
20
+
21
+ local.restore!(downloaded_dir)
22
+ end
23
+ end
24
+ end
25
+ end
@@ -0,0 +1,5 @@
1
+ module Capistrano
2
+ module DBSync
3
+ VERSION = "0.0.2"
4
+ end
5
+ end
@@ -0,0 +1,7 @@
1
+ module Capistrano
2
+ module DBSync
3
+ end
4
+ end
5
+
6
+ Dir.glob(File.join(File.dirname(__FILE__), "/db_sync/**/*.rb")).sort.each { |f| require f }
7
+ load File.join(File.dirname(__FILE__), "/db_sync/tasks.rake")
@@ -0,0 +1,30 @@
1
+ require 'spec_helper'
2
+
3
+ describe Capistrano::DBSync::Postgres::CLI do
4
+ let(:config) do
5
+ {
6
+ "database" => "staging",
7
+ "username" => "user",
8
+ "password" => "pw",
9
+ "adapter" => "postgresql",
10
+ "host" => "127.0.0.1",
11
+ "port" => "5432",
12
+ }
13
+ end
14
+
15
+ let(:cli) { Capistrano::DBSync::Postgres::CLI.new(config) }
16
+
17
+ describe "#dump" do
18
+ it "generates pg_dump command" do
19
+ command = cli.dump("/tmp/staging.dump", "staging", ["--section=pre-data"])
20
+ command.must_equal "PGPASSWORD='pw' pg_dump -U user -h 127.0.0.1 -p 5432 --no-acl --no-owner --format=custom -f /tmp/staging.dump --section=pre-data staging"
21
+ end
22
+ end
23
+
24
+ describe "#restore" do
25
+ it "generates pg_dump command" do
26
+ command = cli.restore("/db/production.dump", "staging", ["--jobs=3"])
27
+ command.must_equal "PGPASSWORD='pw' pg_restore -U user -h 127.0.0.1 -p 5432 --no-acl --no-owner --format=custom -d staging --jobs=3 /db/production.dump"
28
+ end
29
+ end
30
+ end
@@ -0,0 +1,31 @@
1
+ require 'spec_helper'
2
+
3
+ describe Capistrano::DBSync::Postgres::Exporter do
4
+ let(:working_dir) { "/tmp/dumps/" }
5
+ let(:config) { { "database" => "faceburger_production", "host" => "10.20.30.40" } }
6
+ let(:exporter) { Capistrano::DBSync::Postgres::Exporter.new(working_dir, config) }
7
+
8
+ describe "#dump" do
9
+ let(:data_selection) do
10
+ {
11
+ campaigns: "SELECT * FROM campaigns WHERE date > NOW() - interval '160 days'",
12
+ keywords: "SELECT * FROM keywords WHERE created_at > NOW() - interval '160 days'",
13
+ phone_calls: nil
14
+ }
15
+ end
16
+
17
+ it "restore dump files" do
18
+ commands = exporter.dump(data_selection: data_selection)
19
+
20
+ # Assert dumping database schema with data except for tables specified on data_selection
21
+ commands[0].must_match /pg_dump.* -f \/tmp\/dumps\/0001-faceburger_production\.schema/
22
+ commands[0].must_match /--exclude-table-data="campaigns"/
23
+ commands[0].must_match /--exclude-table-data="keywords"/
24
+ commands[0].must_match /--exclude-table-data="phone_calls"/
25
+
26
+ # Assert dumping data for tables specified on data_selection
27
+ commands[1].must_match /COPY.*campaigns.*\/tmp\/dumps\/0002-campaigns\.table/
28
+ commands[2].must_match /COPY.*keywords.*\/tmp\/dumps\/0003-keywords\.table/
29
+ end
30
+ end
31
+ end
@@ -0,0 +1,43 @@
1
+ require 'spec_helper'
2
+
3
+ describe Capistrano::DBSync::Postgres::Importer do
4
+ let(:working_dir) { "/tmp/dumps/" }
5
+ let(:config) { { "database" => "faceburger_development", "host" => "localhost" } }
6
+ let(:importer) { Capistrano::DBSync::Postgres::Importer.new(working_dir, config) }
7
+
8
+ describe "#restore" do
9
+ before do
10
+ Dir.stubs(:glob).with("/tmp/dumps/*.schema")
11
+ .returns(["/tmp/dumps/0001-faceburger_production.schema"])
12
+
13
+ Dir.stubs(:glob).with("/tmp/dumps/*.table")
14
+ .returns(["/tmp/dumps/0002-campaigns.table", "/tmp/dumps/0003-keywords.table"])
15
+ end
16
+
17
+ it "restore dump files" do
18
+ commands = importer.restore(jobs: 3)
19
+
20
+ # Assert drop and create temporary database
21
+ commands[0].must_match /pg_terminate_backend.*faceburger_development_\d+/m
22
+ commands[1].must_match /DROP DATABASE IF EXISTS.*faceburger_development_\d+/
23
+ commands[2].must_match /CREATE DATABASE.*faceburger_development_\d+/
24
+
25
+ # Assert restore schema definition and data of tables with full data
26
+ commands[3].must_match /pg_restore.*--section=pre-data --section=data --jobs=3/
27
+ commands[3].must_match /pg_restore.* \/tmp\/dumps\/0001-faceburger_production\.schema/
28
+
29
+ # Assert import selective tables data
30
+ commands[4].must_match /COPY campaigns.*\/tmp\/dumps\/0002-campaigns\.table/
31
+ commands[5].must_match /COPY keywords.*\/tmp\/dumps\/0003-keywords\.table/
32
+
33
+ # Assert restore indexes, constraints, triggers and rules
34
+ commands[6].must_match /pg_restore.*--section=post-data --jobs=3/
35
+ commands[6].must_match /pg_restore.* \/tmp\/dumps\/0001-faceburger_production\.schema/
36
+
37
+ # Assert rename the temporary database to target restoring database name
38
+ commands[7].must_match /pg_terminate_backend.*faceburger_development/m
39
+ commands[8].must_match /DROP DATABASE IF EXISTS.*faceburger_development/
40
+ commands[9].must_match /ALTER DATABASE.*faceburger_development_\d+.*RENAME TO.*faceburger_development.*/
41
+ end
42
+ end
43
+ end
@@ -0,0 +1,5 @@
1
+ require 'pry'
2
+ require 'capistrano/db_sync'
3
+ require 'minitest/autorun'
4
+ require 'minitest/pride'
5
+ require 'mocha/mini_test'
metadata ADDED
@@ -0,0 +1,84 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: capistrano-db_sync
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.0.2
5
+ platform: ruby
6
+ authors:
7
+ - Rafael Sales
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2015-03-09 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: capistrano
15
+ requirement: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - ">="
18
+ - !ruby/object:Gem::Version
19
+ version: 3.0.0
20
+ type: :runtime
21
+ prerelease: false
22
+ version_requirements: !ruby/object:Gem::Requirement
23
+ requirements:
24
+ - - ">="
25
+ - !ruby/object:Gem::Version
26
+ version: 3.0.0
27
+ description: Fast download and restore dumps using edge features of Postgres 9.x
28
+ email:
29
+ - rafaelcds@gmail.com
30
+ executables: []
31
+ extensions: []
32
+ extra_rdoc_files: []
33
+ files:
34
+ - ".gitignore"
35
+ - Gemfile
36
+ - LICENSE
37
+ - README.md
38
+ - Rakefile
39
+ - capistrano-db_sync.gemspec
40
+ - lib/capistrano/db_sync.rb
41
+ - lib/capistrano/db_sync/configuration.rb
42
+ - lib/capistrano/db_sync/executor/base.rb
43
+ - lib/capistrano/db_sync/executor/local.rb
44
+ - lib/capistrano/db_sync/executor/remote.rb
45
+ - lib/capistrano/db_sync/postgres.rb
46
+ - lib/capistrano/db_sync/postgres/cli.rb
47
+ - lib/capistrano/db_sync/postgres/exporter.rb
48
+ - lib/capistrano/db_sync/postgres/file_name_generator.rb
49
+ - lib/capistrano/db_sync/postgres/importer.rb
50
+ - lib/capistrano/db_sync/tasks.rake
51
+ - lib/capistrano/db_sync/version.rb
52
+ - spec/lib/capistrano/db_sync/postgres/cli_spec.rb
53
+ - spec/lib/capistrano/db_sync/postgres/exporter_spec.rb
54
+ - spec/lib/capistrano/db_sync/postgres/importer_spec.rb
55
+ - spec/spec_helper.rb
56
+ homepage: https://github.com/rafaelsales/capistrano-db_sync
57
+ licenses:
58
+ - MIT
59
+ metadata: {}
60
+ post_install_message:
61
+ rdoc_options: []
62
+ require_paths:
63
+ - lib
64
+ required_ruby_version: !ruby/object:Gem::Requirement
65
+ requirements:
66
+ - - ">="
67
+ - !ruby/object:Gem::Version
68
+ version: '0'
69
+ required_rubygems_version: !ruby/object:Gem::Requirement
70
+ requirements:
71
+ - - ">="
72
+ - !ruby/object:Gem::Version
73
+ version: '0'
74
+ requirements: []
75
+ rubyforge_project:
76
+ rubygems_version: 2.4.5
77
+ signing_key:
78
+ specification_version: 4
79
+ summary: A capistrano task to import remote Postgres databases
80
+ test_files:
81
+ - spec/lib/capistrano/db_sync/postgres/cli_spec.rb
82
+ - spec/lib/capistrano/db_sync/postgres/exporter_spec.rb
83
+ - spec/lib/capistrano/db_sync/postgres/importer_spec.rb
84
+ - spec/spec_helper.rb