pg_export 0.3.2 → 0.4.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.rubocop.yml +0 -22
- data/.travis.yml +1 -1
- data/README.md +32 -12
- data/bin/pg_export +18 -5
- data/codeclimate.yml +15 -0
- data/lib/pg_export.rb +19 -6
- data/lib/pg_export/configuration.rb +3 -1
- data/lib/pg_export/entities/dump/base.rb +14 -4
- data/lib/pg_export/entities/encrypted_dump.rb +7 -0
- data/lib/pg_export/entities/plain_dump.rb +7 -0
- data/lib/pg_export/errors.rb +1 -0
- data/lib/pg_export/interactive.rb +83 -0
- data/lib/pg_export/interactive/refinements/colourable_string.rb +19 -0
- data/lib/pg_export/services/aes.rb +21 -0
- data/lib/pg_export/services/dump_storage.rb +18 -7
- data/lib/pg_export/services/ftp_service.rb +6 -2
- data/lib/pg_export/services/utils.rb +41 -21
- data/lib/pg_export/version.rb +1 -1
- data/pg_export.gemspec +2 -0
- metadata +22 -4
- data/lib/pg_export/entities/compressed_dump.rb +0 -19
- data/lib/pg_export/entities/sql_dump.rb +0 -19
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 58c84b5c514c736a005d23eb4bf21f68961d88b1
|
4
|
+
data.tar.gz: 1db3a0a02ffabc82a48da0e5213955d8411ab0bc
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 5f90ff4abce09f0d34cdc1dd94b5dd545e2b953e9e4acfd10d3ee30c4134bca68ce6b7bb41428796f65e23e58a709d6338533b5d7bc4ee47cb430368c1810165
|
7
|
+
data.tar.gz: 028ff0080d2068d1bdd1d3cd3fba39ceddff98b2845506817d8749202e04e0b6268e3833654264bb81360ac7f839beba8d5f80cf5673f451574560e7eebf228a
|
data/.rubocop.yml
CHANGED
@@ -3,23 +3,11 @@ AllCops:
|
|
3
3
|
Exclude:
|
4
4
|
- 'spec/spec_helper.rb'
|
5
5
|
|
6
|
-
Style/ClassAndModuleChildren:
|
7
|
-
Enabled: false
|
8
|
-
|
9
6
|
Metrics/LineLength:
|
10
7
|
Max: 200
|
11
8
|
|
12
9
|
Metrics/AbcSize:
|
13
10
|
Max: 20
|
14
|
-
Exclude:
|
15
|
-
- 'lib/pg_export.rb'
|
16
|
-
|
17
|
-
Lint/NestedMethodDefinition:
|
18
|
-
Enabled: false
|
19
|
-
|
20
|
-
Lint/HandleExceptions:
|
21
|
-
Exclude:
|
22
|
-
- 'spec/services/utils_spec.rb'
|
23
11
|
|
24
12
|
Lint/AmbiguousOperator:
|
25
13
|
Exclude:
|
@@ -29,18 +17,8 @@ Style/AlignArray:
|
|
29
17
|
Exclude:
|
30
18
|
- 'spec/configuration_spec.rb'
|
31
19
|
|
32
|
-
Style/GuardClause:
|
33
|
-
Exclude:
|
34
|
-
- '*.gemspec'
|
35
|
-
|
36
|
-
Style/AndOr:
|
37
|
-
Enabled: false
|
38
|
-
|
39
20
|
Style/ParallelAssignment:
|
40
21
|
Enabled: false
|
41
22
|
|
42
23
|
Documentation:
|
43
24
|
Enabled: false
|
44
|
-
|
45
|
-
CaseIndentation:
|
46
|
-
Enabled: false
|
data/.travis.yml
CHANGED
data/README.md
CHANGED
@@ -1,5 +1,9 @@
|
|
1
1
|
# PgExport
|
2
2
|
|
3
|
+
[![Gem Version](https://badge.fury.io/rb/pg_export.svg)](https://badge.fury.io/rb/pg_export)
|
4
|
+
[![Build Status](https://travis-ci.org/maicher/pg_export.svg?branch=master)](https://travis-ci.org/maicher/pg_export)
|
5
|
+
[![Code Climate](https://codeclimate.com/github/maicher/pg_export/badges/gpa.svg)](https://codeclimate.com/github/maicher/pg_export)
|
6
|
+
|
3
7
|
CLI for creating and exporting PostgreSQL dumps to FTP.
|
4
8
|
|
5
9
|
Can be used for backups or synchronizing databases between production and development environments.
|
@@ -8,20 +12,23 @@ Example:
|
|
8
12
|
|
9
13
|
pg_export --database database_name -keep 5
|
10
14
|
|
11
|
-
Above command will perform database dump,
|
15
|
+
Above command will perform database dump, encrypt it, upload it to FTP and remove old dumps from FTP, keeping newest 5.
|
12
16
|
|
13
|
-
FTP connection params are configured by env variables.
|
17
|
+
FTP connection params and encryption key are configured by env variables.
|
14
18
|
|
15
19
|
Features:
|
16
20
|
|
17
|
-
- uses shell command pg_dump
|
21
|
+
- uses shell command `pg_dump` and `pg_restore`
|
18
22
|
- no external gem dependencies
|
23
|
+
- encrypts dumps by OpenSSL
|
19
24
|
- uses ruby tempfiles, so local dumps are garbage collected automatically
|
25
|
+
- easy restoring dumps through interactive mode
|
20
26
|
|
21
27
|
## Dependencies
|
22
28
|
|
23
29
|
* Ruby >= 2.1
|
24
30
|
* $ pg_dump
|
31
|
+
* $ pg_restore
|
25
32
|
|
26
33
|
## Installation
|
27
34
|
|
@@ -47,6 +54,7 @@ Or install it yourself as:
|
|
47
54
|
-d, --database DATABASE [Required] Name of the database to export
|
48
55
|
-k, --keep [KEEP] [Optional] Number of dump files to keep on FTP (default: 10)
|
49
56
|
-t, --timestamped [Optional] Enables log messages with timestamps
|
57
|
+
-i, --interactive Interactive, command line mode, for restoring dumps into databases
|
50
58
|
-h, --help Show this message
|
51
59
|
|
52
60
|
Setting can be verified by running following commands:
|
@@ -55,43 +63,55 @@ Or install it yourself as:
|
|
55
63
|
|
56
64
|
## How to start
|
57
65
|
|
58
|
-
__Step 1.__ Prepare FTP account and put configuration into env variables.
|
66
|
+
__Step 1.__ Prepare FTP account and put configuration into env variables. Dumps will be exported into that location.
|
59
67
|
|
60
|
-
# /etc/
|
68
|
+
# /etc/environment
|
61
69
|
BACKUP_FTP_HOST="yourftp.example.com"
|
62
70
|
BACKUP_FTP_USER="user"
|
63
71
|
BACKUP_FTP_PASSWORD="password"
|
64
72
|
|
65
|
-
|
73
|
+
__Step 2.__ Put dump encryption key into env variable (at least 16 characters). Dumps will be SSL(AES-128-CBC) encrypted using that key.
|
74
|
+
|
75
|
+
# /etc/environment
|
76
|
+
DUMP_ENCRYPTION_KEY="1234567890abcdef"
|
77
|
+
|
78
|
+
Variables cannot include `#` sign, [more info](http://serverfault.com/questions/539730/environment-variable-in-etc-environment-with-pound-hash-sign-in-the-value).
|
66
79
|
|
67
|
-
__Step
|
80
|
+
__Step 3.__ Configure how many dumps should be kept in FTP (optional).
|
68
81
|
|
69
82
|
# /etc/environment
|
70
83
|
KEEP_DUMPS=5 # default: 10
|
71
84
|
|
72
|
-
__Step
|
85
|
+
__Step 4.__ Print the configuration to verify whether env variables has been loaded.
|
73
86
|
|
74
87
|
$ pg_export --configuration
|
75
88
|
=> database:
|
76
89
|
keep_dumps: 5
|
90
|
+
dump_password: k40***
|
77
91
|
ftp_host: yourftp.example.com
|
78
92
|
ftp_user: user
|
79
93
|
ftp_password: pass***
|
80
94
|
|
81
|
-
__Step
|
95
|
+
__Step 5.__ Try connecting to FTP to verify the connection.
|
82
96
|
|
83
97
|
$ pg_export --ftp
|
84
98
|
=> Connect to yourftp.example.com
|
85
99
|
Close FTP
|
86
100
|
|
87
|
-
__Step
|
101
|
+
__Step 6.__ Perform database export.
|
88
102
|
|
89
103
|
$ pg_export -d your_database
|
90
104
|
=> Create Dump Tempfile (1.36MB)
|
91
|
-
Create
|
105
|
+
Create Encrypted Dump Tempfile (1.34MB)
|
92
106
|
Connect to yourftp.example.com
|
93
|
-
Export
|
107
|
+
Export Encrypted Dump Tempfile (1.34MB) your_database_20161020_125747 to yourftp.example.com
|
94
108
|
Close FTP
|
109
|
+
|
110
|
+
## How to restore a dump?
|
111
|
+
|
112
|
+
Go to interactive mode and follow the instructions:
|
113
|
+
|
114
|
+
pg_export -i
|
95
115
|
|
96
116
|
## Development
|
97
117
|
|
data/bin/pg_export
CHANGED
@@ -3,9 +3,13 @@
|
|
3
3
|
require 'optparse'
|
4
4
|
require 'ostruct'
|
5
5
|
require 'pg_export'
|
6
|
+
require 'pg_export/interactive'
|
6
7
|
|
8
|
+
LOGS_DEFAULT = ->(_, _, _, message) { "#{message}\n" }
|
7
9
|
LOGS_TIMESTAMPED = ->(severity, datetime, progname, message) { "#{datetime} #{Process.pid} TID-#{Thread.current.object_id.to_s(36)}#{progname} #{severity}: #{message}\n" }
|
8
|
-
|
10
|
+
LOGS_MUTED = ->(_, _, _, _) {}
|
11
|
+
|
12
|
+
PgExport::Logging.logger.formatter = LOGS_DEFAULT
|
9
13
|
|
10
14
|
options = OpenStruct.new
|
11
15
|
option_parser = OptionParser.new do |opts|
|
@@ -23,6 +27,10 @@ option_parser = OptionParser.new do |opts|
|
|
23
27
|
options.timestamped = true
|
24
28
|
end
|
25
29
|
|
30
|
+
opts.on('-i', '--interactive', 'Interactive, command line mode, for restoring dumps into databases') do
|
31
|
+
options.interactive = true
|
32
|
+
end
|
33
|
+
|
26
34
|
opts.on('-h', '--help', 'Show this message') do
|
27
35
|
puts opts
|
28
36
|
exit
|
@@ -36,7 +44,6 @@ option_parser = OptionParser.new do |opts|
|
|
36
44
|
end
|
37
45
|
|
38
46
|
opts.on('-f', '--ftp', 'Tries connecting to FTP to verify the connection') do
|
39
|
-
PgExport::Logging.logger.formatter = LOGS_NORMAL
|
40
47
|
PgExport::FtpService.new(PgExport::Configuration.new.ftp_params)
|
41
48
|
exit
|
42
49
|
end
|
@@ -45,12 +52,18 @@ end
|
|
45
52
|
begin
|
46
53
|
option_parser.parse!
|
47
54
|
|
48
|
-
|
55
|
+
options.database ||= 'undefined' if options.interactive
|
56
|
+
PgExport::Logging.logger.formatter = LOGS_TIMESTAMPED if options.timestamped
|
57
|
+
PgExport::Logging.logger.formatter = LOGS_MUTED if options.interactive
|
49
58
|
|
50
|
-
PgExport.new do |config|
|
59
|
+
pg_export = PgExport.new do |config|
|
51
60
|
config.database = options.database if options.database
|
52
61
|
config.keep_dumps = options.keep if options.keep
|
53
|
-
end
|
62
|
+
end
|
63
|
+
pg_export.extend(PgExport::Interactive) if options.interactive
|
64
|
+
|
65
|
+
pg_export.call
|
66
|
+
|
54
67
|
rescue OptionParser::MissingArgument, PgExport::InvalidConfigurationError => e
|
55
68
|
puts "Error: #{e}"
|
56
69
|
puts option_parser
|
data/codeclimate.yml
ADDED
data/lib/pg_export.rb
CHANGED
@@ -2,7 +2,11 @@ require 'logger'
|
|
2
2
|
require 'tempfile'
|
3
3
|
require 'zlib'
|
4
4
|
require 'net/ftp'
|
5
|
+
require 'openssl'
|
5
6
|
require 'forwardable'
|
7
|
+
require 'open3'
|
8
|
+
|
9
|
+
require 'cli_spinnable'
|
6
10
|
|
7
11
|
require 'pg_export/version'
|
8
12
|
require 'pg_export/logging'
|
@@ -11,12 +15,13 @@ require 'pg_export/configuration'
|
|
11
15
|
require 'pg_export/concurrency'
|
12
16
|
require 'pg_export/entities/dump/size_human'
|
13
17
|
require 'pg_export/entities/dump/base'
|
14
|
-
require 'pg_export/entities/
|
15
|
-
require 'pg_export/entities/
|
18
|
+
require 'pg_export/entities/plain_dump'
|
19
|
+
require 'pg_export/entities/encrypted_dump'
|
16
20
|
require 'pg_export/services/ftp_service'
|
17
21
|
require 'pg_export/services/ftp_service/connection'
|
18
22
|
require 'pg_export/services/utils'
|
19
23
|
require 'pg_export/services/dump_storage'
|
24
|
+
require 'pg_export/services/aes'
|
20
25
|
|
21
26
|
class PgExport
|
22
27
|
include Concurrency
|
@@ -42,14 +47,22 @@ class PgExport
|
|
42
47
|
attr_reader :config
|
43
48
|
attr_accessor :dump, :dump_storage
|
44
49
|
|
50
|
+
def create_dump
|
51
|
+
sql_dump = utils.create_dump(config.database)
|
52
|
+
enc_dump = utils.encrypt(sql_dump)
|
53
|
+
self.dump = enc_dump
|
54
|
+
end
|
55
|
+
|
45
56
|
def initialize_dump_storage
|
46
57
|
ftp_service = FtpService.new(config.ftp_params)
|
47
58
|
self.dump_storage = DumpStorage.new(ftp_service, config.database)
|
59
|
+
self
|
48
60
|
end
|
49
61
|
|
50
|
-
def
|
51
|
-
|
52
|
-
|
53
|
-
|
62
|
+
def utils
|
63
|
+
@utils ||= Utils.new(
|
64
|
+
Aes.encryptor(config.dump_encryption_key),
|
65
|
+
Aes.decryptor(config.dump_encryption_key)
|
66
|
+
)
|
54
67
|
end
|
55
68
|
end
|
@@ -3,6 +3,7 @@ class PgExport
|
|
3
3
|
DEFAULTS = {
|
4
4
|
database: nil,
|
5
5
|
keep_dumps: ENV['KEEP_DUMPS'] || 10,
|
6
|
+
dump_encryption_key: ENV['DUMP_ENCRYPTION_KEY'],
|
6
7
|
ftp_host: ENV['BACKUP_FTP_HOST'],
|
7
8
|
ftp_user: ENV['BACKUP_FTP_USER'],
|
8
9
|
ftp_password: ENV['BACKUP_FTP_PASSWORD']
|
@@ -20,6 +21,7 @@ class PgExport
|
|
20
21
|
DEFAULTS.keys.each do |field|
|
21
22
|
raise InvalidConfigurationError, "Field #{field} is required" if send(field).nil?
|
22
23
|
end
|
24
|
+
raise InvalidConfigurationError, 'Dump password is to short. It should have at least 16 characters' if dump_encryption_key.length < 16
|
23
25
|
end
|
24
26
|
|
25
27
|
def ftp_params
|
@@ -37,7 +39,7 @@ class PgExport
|
|
37
39
|
private
|
38
40
|
|
39
41
|
def print_attr(key)
|
40
|
-
if key
|
42
|
+
if %i(ftp_password dump_encryption_key).include?(key)
|
41
43
|
if send(key)
|
42
44
|
"#{key}: #{send(key)[0..2]}***\n"
|
43
45
|
else
|
@@ -6,18 +6,28 @@ class PgExport
|
|
6
6
|
|
7
7
|
CHUNK_SIZE = (2**16).freeze
|
8
8
|
|
9
|
-
def_delegators :file, :path, :read, :write, :rewind, :size, :eof?
|
9
|
+
def_delegators :file, :path, :read, :write, :<<, :rewind, :close, :size, :eof?
|
10
10
|
|
11
11
|
def initialize
|
12
12
|
@file = Tempfile.new('dump')
|
13
13
|
end
|
14
14
|
|
15
15
|
def ext
|
16
|
-
|
16
|
+
''
|
17
17
|
end
|
18
18
|
|
19
|
-
def open
|
20
|
-
|
19
|
+
def open(operation_type, &block)
|
20
|
+
case operation_type.to_sym
|
21
|
+
when :read then File.open(path, 'r', &block)
|
22
|
+
when :write then File.open(path, 'w', &block)
|
23
|
+
else raise ArgumentError, 'Operation type can be only :read or :write'
|
24
|
+
end
|
25
|
+
end
|
26
|
+
|
27
|
+
def each_chunk
|
28
|
+
open(:read) do |file|
|
29
|
+
yield file.read(CHUNK_SIZE) until file.eof?
|
30
|
+
end
|
21
31
|
end
|
22
32
|
|
23
33
|
def to_s
|
data/lib/pg_export/errors.rb
CHANGED
@@ -0,0 +1,83 @@
|
|
1
|
+
require 'pg_export/interactive/refinements/colourable_string'
|
2
|
+
|
3
|
+
class PgExport
|
4
|
+
module Interactive
|
5
|
+
include CliSpinnable
|
6
|
+
using ColourableString
|
7
|
+
|
8
|
+
def self.extended(_)
|
9
|
+
puts 'Interactive mode, for restoring dump into database.'.green
|
10
|
+
end
|
11
|
+
|
12
|
+
def call
|
13
|
+
initialize_dump_storage
|
14
|
+
print_all_dumps
|
15
|
+
download_selected_dump
|
16
|
+
restore_downloaded_dump
|
17
|
+
puts 'Success'.green
|
18
|
+
self
|
19
|
+
end
|
20
|
+
|
21
|
+
private
|
22
|
+
|
23
|
+
def initialize_dump_storage
|
24
|
+
with_spinner do |cli|
|
25
|
+
cli.print 'Connecting to FTP'
|
26
|
+
super
|
27
|
+
cli.tick
|
28
|
+
end
|
29
|
+
end
|
30
|
+
|
31
|
+
def print_all_dumps
|
32
|
+
dumps.each.with_index(0) do |name, i|
|
33
|
+
print "(#{i}) "
|
34
|
+
puts name.to_s.gray
|
35
|
+
end
|
36
|
+
self
|
37
|
+
end
|
38
|
+
|
39
|
+
def download_selected_dump
|
40
|
+
puts 'Which dump would you like to import?'
|
41
|
+
print "Type from 1 to #{dumps.count - 1} (0): "
|
42
|
+
name = dumps.fetch(gets.chomp.to_i)
|
43
|
+
with_spinner do |cli|
|
44
|
+
cli.print 'Downloading dump'
|
45
|
+
encrypted_dump = dump_storage.download(name)
|
46
|
+
cli.print " (#{encrypted_dump.size_human})"
|
47
|
+
cli.tick
|
48
|
+
cli.print 'Decrypting dump'
|
49
|
+
self.dump = utils.decrypt(encrypted_dump)
|
50
|
+
cli.print " (#{dump.size_human})"
|
51
|
+
cli.tick
|
52
|
+
end
|
53
|
+
self
|
54
|
+
rescue OpenSSL::Cipher::CipherError => e
|
55
|
+
puts "Problem decrypting dump file: #{e}. Try again.".red
|
56
|
+
retry
|
57
|
+
end
|
58
|
+
|
59
|
+
def restore_downloaded_dump
|
60
|
+
puts 'To which database you would like to restore the downloaded dump?'
|
61
|
+
if config.database == 'undefined'
|
62
|
+
print 'Enter a local database name: '
|
63
|
+
else
|
64
|
+
print "Enter a local database name (#{config.database}): "
|
65
|
+
end
|
66
|
+
database = gets.chomp
|
67
|
+
database = database.empty? ? config.database : database
|
68
|
+
with_spinner do |cli|
|
69
|
+
cli.print "Restoring dump to #{database} database"
|
70
|
+
utils.restore_dump(dump, database)
|
71
|
+
cli.tick
|
72
|
+
end
|
73
|
+
self
|
74
|
+
rescue PgRestoreError => e
|
75
|
+
puts e.to_s.red
|
76
|
+
retry
|
77
|
+
end
|
78
|
+
|
79
|
+
def dumps
|
80
|
+
@dumps ||= dump_storage.all
|
81
|
+
end
|
82
|
+
end
|
83
|
+
end
|
@@ -0,0 +1,19 @@
|
|
1
|
+
class PgExport
|
2
|
+
module Interactive
|
3
|
+
module ColourableString
|
4
|
+
refine String do
|
5
|
+
def red
|
6
|
+
"\e[31m#{self}\e[0m"
|
7
|
+
end
|
8
|
+
|
9
|
+
def green
|
10
|
+
"\e[0;32;49m#{self}\e[0m"
|
11
|
+
end
|
12
|
+
|
13
|
+
def gray
|
14
|
+
"\e[37m#{self}\e[0m"
|
15
|
+
end
|
16
|
+
end
|
17
|
+
end
|
18
|
+
end
|
19
|
+
end
|
@@ -0,0 +1,21 @@
|
|
1
|
+
class PgExport
|
2
|
+
class Aes
|
3
|
+
ALG = 'AES-128-CBC'.freeze
|
4
|
+
|
5
|
+
def self.encryptor(key)
|
6
|
+
initialize_aes(:encrypt, key)
|
7
|
+
end
|
8
|
+
|
9
|
+
def self.decryptor(key)
|
10
|
+
initialize_aes(:decrypt, key)
|
11
|
+
end
|
12
|
+
|
13
|
+
def self.initialize_aes(mode, key)
|
14
|
+
raise ArgumentError, 'Only :encrypt or :decrypt are allowed' unless %i(encrypt decrypt).include?(mode)
|
15
|
+
aes = OpenSSL::Cipher::Cipher.new(ALG)
|
16
|
+
aes.public_send(mode.to_sym)
|
17
|
+
aes.key = key
|
18
|
+
aes
|
19
|
+
end
|
20
|
+
end
|
21
|
+
end
|
@@ -12,16 +12,31 @@ class PgExport
|
|
12
12
|
def upload(dump)
|
13
13
|
dump_name = timestamped_name(dump)
|
14
14
|
ftp_service.upload_file(dump.path, dump_name)
|
15
|
-
logger.info "
|
15
|
+
logger.info "Upload #{dump} #{dump_name} to #{ftp_service}"
|
16
|
+
end
|
17
|
+
|
18
|
+
def download(name)
|
19
|
+
dump = EncryptedDump.new
|
20
|
+
ftp_service.download_file(dump.path, name)
|
21
|
+
logger.info "Download #{dump} #{name} from #{ftp_service}"
|
22
|
+
dump
|
16
23
|
end
|
17
24
|
|
18
25
|
def remove_old(keep:)
|
19
|
-
|
26
|
+
find_by_name(name).drop(keep.to_i).each do |filename|
|
20
27
|
ftp_service.delete(filename)
|
21
|
-
logger.info "Remove #{filename} from
|
28
|
+
logger.info "Remove #{filename} from #{ftp_service}"
|
22
29
|
end
|
23
30
|
end
|
24
31
|
|
32
|
+
def find_by_name(s)
|
33
|
+
ftp_service.list(s + '_*')
|
34
|
+
end
|
35
|
+
|
36
|
+
def all
|
37
|
+
ftp_service.list('*')
|
38
|
+
end
|
39
|
+
|
25
40
|
private
|
26
41
|
|
27
42
|
attr_reader :ftp_service, :name
|
@@ -29,9 +44,5 @@ class PgExport
|
|
29
44
|
def timestamped_name(dump)
|
30
45
|
name + Time.now.strftime(TIMESTAMP) + dump.ext
|
31
46
|
end
|
32
|
-
|
33
|
-
def ftp_regex
|
34
|
-
name + '_*'
|
35
|
-
end
|
36
47
|
end
|
37
48
|
end
|
@@ -9,8 +9,8 @@ class PgExport
|
|
9
9
|
ObjectSpace.define_finalizer(self, proc { connection.close })
|
10
10
|
end
|
11
11
|
|
12
|
-
def list(
|
13
|
-
ftp.list(
|
12
|
+
def list(regex_string)
|
13
|
+
ftp.list(regex_string).map { |item| item.split(' ').last }.sort.reverse
|
14
14
|
end
|
15
15
|
|
16
16
|
def delete(filename)
|
@@ -21,6 +21,10 @@ class PgExport
|
|
21
21
|
ftp.putbinaryfile(path.to_s, name, CHUNK_SIZE)
|
22
22
|
end
|
23
23
|
|
24
|
+
def download_file(path, name)
|
25
|
+
ftp.getbinaryfile(name, path.to_s, CHUNK_SIZE)
|
26
|
+
end
|
27
|
+
|
24
28
|
def to_s
|
25
29
|
host
|
26
30
|
end
|
@@ -1,37 +1,57 @@
|
|
1
1
|
class PgExport
|
2
2
|
class Utils
|
3
|
-
|
3
|
+
include Logging
|
4
4
|
|
5
|
-
def
|
6
|
-
|
7
|
-
|
8
|
-
|
5
|
+
def initialize(encryptor, decryptor)
|
6
|
+
@encryptor, @decryptor = encryptor, decryptor
|
7
|
+
end
|
8
|
+
|
9
|
+
def create_dump(database_name)
|
10
|
+
dump = PlainDump.new
|
11
|
+
Open3.popen3("pg_dump -Fc --file #{dump.path} #{database_name}") do |_, _, err|
|
12
|
+
error = err.read
|
13
|
+
raise PgDumpError, error unless error.empty?
|
14
|
+
end
|
9
15
|
logger.info "Create #{dump}"
|
10
16
|
dump
|
11
17
|
end
|
12
18
|
|
13
|
-
def
|
14
|
-
|
15
|
-
|
16
|
-
|
17
|
-
gz.write(f.read(Dump::Base::CHUNK_SIZE)) until f.eof?
|
18
|
-
end
|
19
|
+
def restore_dump(dump, database_name)
|
20
|
+
Open3.popen3("pg_restore -c -d #{database_name} #{dump.path}") do |_, _, err|
|
21
|
+
error = err.read
|
22
|
+
raise PgRestoreError, error if /FATAL/ =~ error
|
19
23
|
end
|
20
|
-
|
21
|
-
|
22
|
-
dump_gz
|
24
|
+
logger.info "Restore #{dump}"
|
25
|
+
self
|
23
26
|
end
|
24
27
|
|
25
|
-
def
|
26
|
-
|
27
|
-
|
28
|
-
|
29
|
-
|
30
|
-
|
31
|
-
end
|
28
|
+
def encrypt(dump)
|
29
|
+
enc_dump = EncryptedDump.new
|
30
|
+
copy_using(encryptor, from: dump, to: enc_dump)
|
31
|
+
logger.info "Create #{enc_dump}"
|
32
|
+
enc_dump
|
33
|
+
end
|
32
34
|
|
35
|
+
def decrypt(enc_dump)
|
36
|
+
dump = PlainDump.new
|
37
|
+
copy_using(decryptor, from: enc_dump, to: dump)
|
33
38
|
logger.info "Create #{dump}"
|
34
39
|
dump
|
35
40
|
end
|
41
|
+
|
42
|
+
private
|
43
|
+
|
44
|
+
attr_reader :encryptor, :decryptor
|
45
|
+
|
46
|
+
def copy_using(aes, from:, to:)
|
47
|
+
aes.reset
|
48
|
+
to.open(:write) do |f|
|
49
|
+
from.each_chunk do |chunk|
|
50
|
+
f << aes.update(chunk)
|
51
|
+
end
|
52
|
+
f << aes.final
|
53
|
+
end
|
54
|
+
self
|
55
|
+
end
|
36
56
|
end
|
37
57
|
end
|
data/lib/pg_export/version.rb
CHANGED
data/pg_export.gemspec
CHANGED
@@ -20,6 +20,8 @@ Gem::Specification.new do |spec|
|
|
20
20
|
spec.require_paths = ['lib']
|
21
21
|
spec.required_ruby_version = '>= 2.1.0'
|
22
22
|
|
23
|
+
spec.add_dependency 'cli_spinnable', '~> 0.2'
|
24
|
+
|
23
25
|
spec.add_development_dependency 'bundler', '~> 1.10'
|
24
26
|
spec.add_development_dependency 'rubocop', '~> 0.44'
|
25
27
|
spec.add_development_dependency 'rake', '~> 10.0'
|
metadata
CHANGED
@@ -1,15 +1,29 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: pg_export
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.4.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Krzysztof Maicher
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2016-
|
11
|
+
date: 2016-11-24 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
|
+
- !ruby/object:Gem::Dependency
|
14
|
+
name: cli_spinnable
|
15
|
+
requirement: !ruby/object:Gem::Requirement
|
16
|
+
requirements:
|
17
|
+
- - "~>"
|
18
|
+
- !ruby/object:Gem::Version
|
19
|
+
version: '0.2'
|
20
|
+
type: :runtime
|
21
|
+
prerelease: false
|
22
|
+
version_requirements: !ruby/object:Gem::Requirement
|
23
|
+
requirements:
|
24
|
+
- - "~>"
|
25
|
+
- !ruby/object:Gem::Version
|
26
|
+
version: '0.2'
|
13
27
|
- !ruby/object:Gem::Dependency
|
14
28
|
name: bundler
|
15
29
|
requirement: !ruby/object:Gem::Requirement
|
@@ -116,15 +130,19 @@ files:
|
|
116
130
|
- bin/console
|
117
131
|
- bin/pg_export
|
118
132
|
- bin/setup
|
133
|
+
- codeclimate.yml
|
119
134
|
- lib/pg_export.rb
|
120
135
|
- lib/pg_export/concurrency.rb
|
121
136
|
- lib/pg_export/configuration.rb
|
122
|
-
- lib/pg_export/entities/compressed_dump.rb
|
123
137
|
- lib/pg_export/entities/dump/base.rb
|
124
138
|
- lib/pg_export/entities/dump/size_human.rb
|
125
|
-
- lib/pg_export/entities/
|
139
|
+
- lib/pg_export/entities/encrypted_dump.rb
|
140
|
+
- lib/pg_export/entities/plain_dump.rb
|
126
141
|
- lib/pg_export/errors.rb
|
142
|
+
- lib/pg_export/interactive.rb
|
143
|
+
- lib/pg_export/interactive/refinements/colourable_string.rb
|
127
144
|
- lib/pg_export/logging.rb
|
145
|
+
- lib/pg_export/services/aes.rb
|
128
146
|
- lib/pg_export/services/dump_storage.rb
|
129
147
|
- lib/pg_export/services/ftp_service.rb
|
130
148
|
- lib/pg_export/services/ftp_service/connection.rb
|
@@ -1,19 +0,0 @@
|
|
1
|
-
class PgExport
|
2
|
-
class CompressedDump < Dump::Base
|
3
|
-
def name
|
4
|
-
'Compressed Dump'
|
5
|
-
end
|
6
|
-
|
7
|
-
def ext
|
8
|
-
'.gz'
|
9
|
-
end
|
10
|
-
|
11
|
-
def open(operation_type, &block)
|
12
|
-
case operation_type.to_sym
|
13
|
-
when :read then Zlib::GzipReader.open(path, &block)
|
14
|
-
when :write then Zlib::GzipWriter.open(path, &block)
|
15
|
-
else raise ArgumentError, 'Operation type can be only :read or :write'
|
16
|
-
end
|
17
|
-
end
|
18
|
-
end
|
19
|
-
end
|
@@ -1,19 +0,0 @@
|
|
1
|
-
class PgExport
|
2
|
-
class SqlDump < Dump::Base
|
3
|
-
def name
|
4
|
-
'Dump'
|
5
|
-
end
|
6
|
-
|
7
|
-
def ext
|
8
|
-
''
|
9
|
-
end
|
10
|
-
|
11
|
-
def open(operation_type, &block)
|
12
|
-
case operation_type.to_sym
|
13
|
-
when :read then File.open(path, 'r', &block)
|
14
|
-
when :write then File.open(path, 'w', &block)
|
15
|
-
else raise ArgumentError, 'Operation type can be only :read or :write'
|
16
|
-
end
|
17
|
-
end
|
18
|
-
end
|
19
|
-
end
|