pg_export 0.3.2 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.rubocop.yml +0 -22
- data/.travis.yml +1 -1
- data/README.md +32 -12
- data/bin/pg_export +18 -5
- data/codeclimate.yml +15 -0
- data/lib/pg_export.rb +19 -6
- data/lib/pg_export/configuration.rb +3 -1
- data/lib/pg_export/entities/dump/base.rb +14 -4
- data/lib/pg_export/entities/encrypted_dump.rb +7 -0
- data/lib/pg_export/entities/plain_dump.rb +7 -0
- data/lib/pg_export/errors.rb +1 -0
- data/lib/pg_export/interactive.rb +83 -0
- data/lib/pg_export/interactive/refinements/colourable_string.rb +19 -0
- data/lib/pg_export/services/aes.rb +21 -0
- data/lib/pg_export/services/dump_storage.rb +18 -7
- data/lib/pg_export/services/ftp_service.rb +6 -2
- data/lib/pg_export/services/utils.rb +41 -21
- data/lib/pg_export/version.rb +1 -1
- data/pg_export.gemspec +2 -0
- metadata +22 -4
- data/lib/pg_export/entities/compressed_dump.rb +0 -19
- data/lib/pg_export/entities/sql_dump.rb +0 -19
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 58c84b5c514c736a005d23eb4bf21f68961d88b1
|
4
|
+
data.tar.gz: 1db3a0a02ffabc82a48da0e5213955d8411ab0bc
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 5f90ff4abce09f0d34cdc1dd94b5dd545e2b953e9e4acfd10d3ee30c4134bca68ce6b7bb41428796f65e23e58a709d6338533b5d7bc4ee47cb430368c1810165
|
7
|
+
data.tar.gz: 028ff0080d2068d1bdd1d3cd3fba39ceddff98b2845506817d8749202e04e0b6268e3833654264bb81360ac7f839beba8d5f80cf5673f451574560e7eebf228a
|
data/.rubocop.yml
CHANGED
@@ -3,23 +3,11 @@ AllCops:
|
|
3
3
|
Exclude:
|
4
4
|
- 'spec/spec_helper.rb'
|
5
5
|
|
6
|
-
Style/ClassAndModuleChildren:
|
7
|
-
Enabled: false
|
8
|
-
|
9
6
|
Metrics/LineLength:
|
10
7
|
Max: 200
|
11
8
|
|
12
9
|
Metrics/AbcSize:
|
13
10
|
Max: 20
|
14
|
-
Exclude:
|
15
|
-
- 'lib/pg_export.rb'
|
16
|
-
|
17
|
-
Lint/NestedMethodDefinition:
|
18
|
-
Enabled: false
|
19
|
-
|
20
|
-
Lint/HandleExceptions:
|
21
|
-
Exclude:
|
22
|
-
- 'spec/services/utils_spec.rb'
|
23
11
|
|
24
12
|
Lint/AmbiguousOperator:
|
25
13
|
Exclude:
|
@@ -29,18 +17,8 @@ Style/AlignArray:
|
|
29
17
|
Exclude:
|
30
18
|
- 'spec/configuration_spec.rb'
|
31
19
|
|
32
|
-
Style/GuardClause:
|
33
|
-
Exclude:
|
34
|
-
- '*.gemspec'
|
35
|
-
|
36
|
-
Style/AndOr:
|
37
|
-
Enabled: false
|
38
|
-
|
39
20
|
Style/ParallelAssignment:
|
40
21
|
Enabled: false
|
41
22
|
|
42
23
|
Documentation:
|
43
24
|
Enabled: false
|
44
|
-
|
45
|
-
CaseIndentation:
|
46
|
-
Enabled: false
|
data/.travis.yml
CHANGED
data/README.md
CHANGED
@@ -1,5 +1,9 @@
|
|
1
1
|
# PgExport
|
2
2
|
|
3
|
+
[](https://badge.fury.io/rb/pg_export)
|
4
|
+
[](https://travis-ci.org/maicher/pg_export)
|
5
|
+
[](https://codeclimate.com/github/maicher/pg_export)
|
6
|
+
|
3
7
|
CLI for creating and exporting PostgreSQL dumps to FTP.
|
4
8
|
|
5
9
|
Can be used for backups or synchronizing databases between production and development environments.
|
@@ -8,20 +12,23 @@ Example:
|
|
8
12
|
|
9
13
|
pg_export --database database_name -keep 5
|
10
14
|
|
11
|
-
Above command will perform database dump,
|
15
|
+
Above command will perform database dump, encrypt it, upload it to FTP and remove old dumps from FTP, keeping newest 5.
|
12
16
|
|
13
|
-
FTP connection params are configured by env variables.
|
17
|
+
FTP connection params and encryption key are configured by env variables.
|
14
18
|
|
15
19
|
Features:
|
16
20
|
|
17
|
-
- uses shell command pg_dump
|
21
|
+
- uses shell command `pg_dump` and `pg_restore`
|
18
22
|
- no external gem dependencies
|
23
|
+
- encrypts dumps by OpenSSL
|
19
24
|
- uses ruby tempfiles, so local dumps are garbage collected automatically
|
25
|
+
- easy restoring dumps through interactive mode
|
20
26
|
|
21
27
|
## Dependencies
|
22
28
|
|
23
29
|
* Ruby >= 2.1
|
24
30
|
* $ pg_dump
|
31
|
+
* $ pg_restore
|
25
32
|
|
26
33
|
## Installation
|
27
34
|
|
@@ -47,6 +54,7 @@ Or install it yourself as:
|
|
47
54
|
-d, --database DATABASE [Required] Name of the database to export
|
48
55
|
-k, --keep [KEEP] [Optional] Number of dump files to keep on FTP (default: 10)
|
49
56
|
-t, --timestamped [Optional] Enables log messages with timestamps
|
57
|
+
-i, --interactive Interactive, command line mode, for restoring dumps into databases
|
50
58
|
-h, --help Show this message
|
51
59
|
|
52
60
|
Setting can be verified by running following commands:
|
@@ -55,43 +63,55 @@ Or install it yourself as:
|
|
55
63
|
|
56
64
|
## How to start
|
57
65
|
|
58
|
-
__Step 1.__ Prepare FTP account and put configuration into env variables.
|
66
|
+
__Step 1.__ Prepare FTP account and put configuration into env variables. Dumps will be exported into that location.
|
59
67
|
|
60
|
-
# /etc/
|
68
|
+
# /etc/environment
|
61
69
|
BACKUP_FTP_HOST="yourftp.example.com"
|
62
70
|
BACKUP_FTP_USER="user"
|
63
71
|
BACKUP_FTP_PASSWORD="password"
|
64
72
|
|
65
|
-
|
73
|
+
__Step 2.__ Put dump encryption key into env variable (at least 16 characters). Dumps will be SSL(AES-128-CBC) encrypted using that key.
|
74
|
+
|
75
|
+
# /etc/environment
|
76
|
+
DUMP_ENCRYPTION_KEY="1234567890abcdef"
|
77
|
+
|
78
|
+
Variables cannot include `#` sign, [more info](http://serverfault.com/questions/539730/environment-variable-in-etc-environment-with-pound-hash-sign-in-the-value).
|
66
79
|
|
67
|
-
__Step
|
80
|
+
__Step 3.__ Configure how many dumps should be kept in FTP (optional).
|
68
81
|
|
69
82
|
# /etc/environment
|
70
83
|
KEEP_DUMPS=5 # default: 10
|
71
84
|
|
72
|
-
__Step
|
85
|
+
__Step 4.__ Print the configuration to verify whether env variables has been loaded.
|
73
86
|
|
74
87
|
$ pg_export --configuration
|
75
88
|
=> database:
|
76
89
|
keep_dumps: 5
|
90
|
+
dump_password: k40***
|
77
91
|
ftp_host: yourftp.example.com
|
78
92
|
ftp_user: user
|
79
93
|
ftp_password: pass***
|
80
94
|
|
81
|
-
__Step
|
95
|
+
__Step 5.__ Try connecting to FTP to verify the connection.
|
82
96
|
|
83
97
|
$ pg_export --ftp
|
84
98
|
=> Connect to yourftp.example.com
|
85
99
|
Close FTP
|
86
100
|
|
87
|
-
__Step
|
101
|
+
__Step 6.__ Perform database export.
|
88
102
|
|
89
103
|
$ pg_export -d your_database
|
90
104
|
=> Create Dump Tempfile (1.36MB)
|
91
|
-
Create
|
105
|
+
Create Encrypted Dump Tempfile (1.34MB)
|
92
106
|
Connect to yourftp.example.com
|
93
|
-
Export
|
107
|
+
Export Encrypted Dump Tempfile (1.34MB) your_database_20161020_125747 to yourftp.example.com
|
94
108
|
Close FTP
|
109
|
+
|
110
|
+
## How to restore a dump?
|
111
|
+
|
112
|
+
Go to interactive mode and follow the instructions:
|
113
|
+
|
114
|
+
pg_export -i
|
95
115
|
|
96
116
|
## Development
|
97
117
|
|
data/bin/pg_export
CHANGED
@@ -3,9 +3,13 @@
|
|
3
3
|
require 'optparse'
|
4
4
|
require 'ostruct'
|
5
5
|
require 'pg_export'
|
6
|
+
require 'pg_export/interactive'
|
6
7
|
|
8
|
+
LOGS_DEFAULT = ->(_, _, _, message) { "#{message}\n" }
|
7
9
|
LOGS_TIMESTAMPED = ->(severity, datetime, progname, message) { "#{datetime} #{Process.pid} TID-#{Thread.current.object_id.to_s(36)}#{progname} #{severity}: #{message}\n" }
|
8
|
-
|
10
|
+
LOGS_MUTED = ->(_, _, _, _) {}
|
11
|
+
|
12
|
+
PgExport::Logging.logger.formatter = LOGS_DEFAULT
|
9
13
|
|
10
14
|
options = OpenStruct.new
|
11
15
|
option_parser = OptionParser.new do |opts|
|
@@ -23,6 +27,10 @@ option_parser = OptionParser.new do |opts|
|
|
23
27
|
options.timestamped = true
|
24
28
|
end
|
25
29
|
|
30
|
+
opts.on('-i', '--interactive', 'Interactive, command line mode, for restoring dumps into databases') do
|
31
|
+
options.interactive = true
|
32
|
+
end
|
33
|
+
|
26
34
|
opts.on('-h', '--help', 'Show this message') do
|
27
35
|
puts opts
|
28
36
|
exit
|
@@ -36,7 +44,6 @@ option_parser = OptionParser.new do |opts|
|
|
36
44
|
end
|
37
45
|
|
38
46
|
opts.on('-f', '--ftp', 'Tries connecting to FTP to verify the connection') do
|
39
|
-
PgExport::Logging.logger.formatter = LOGS_NORMAL
|
40
47
|
PgExport::FtpService.new(PgExport::Configuration.new.ftp_params)
|
41
48
|
exit
|
42
49
|
end
|
@@ -45,12 +52,18 @@ end
|
|
45
52
|
begin
|
46
53
|
option_parser.parse!
|
47
54
|
|
48
|
-
|
55
|
+
options.database ||= 'undefined' if options.interactive
|
56
|
+
PgExport::Logging.logger.formatter = LOGS_TIMESTAMPED if options.timestamped
|
57
|
+
PgExport::Logging.logger.formatter = LOGS_MUTED if options.interactive
|
49
58
|
|
50
|
-
PgExport.new do |config|
|
59
|
+
pg_export = PgExport.new do |config|
|
51
60
|
config.database = options.database if options.database
|
52
61
|
config.keep_dumps = options.keep if options.keep
|
53
|
-
end
|
62
|
+
end
|
63
|
+
pg_export.extend(PgExport::Interactive) if options.interactive
|
64
|
+
|
65
|
+
pg_export.call
|
66
|
+
|
54
67
|
rescue OptionParser::MissingArgument, PgExport::InvalidConfigurationError => e
|
55
68
|
puts "Error: #{e}"
|
56
69
|
puts option_parser
|
data/codeclimate.yml
ADDED
data/lib/pg_export.rb
CHANGED
@@ -2,7 +2,11 @@ require 'logger'
|
|
2
2
|
require 'tempfile'
|
3
3
|
require 'zlib'
|
4
4
|
require 'net/ftp'
|
5
|
+
require 'openssl'
|
5
6
|
require 'forwardable'
|
7
|
+
require 'open3'
|
8
|
+
|
9
|
+
require 'cli_spinnable'
|
6
10
|
|
7
11
|
require 'pg_export/version'
|
8
12
|
require 'pg_export/logging'
|
@@ -11,12 +15,13 @@ require 'pg_export/configuration'
|
|
11
15
|
require 'pg_export/concurrency'
|
12
16
|
require 'pg_export/entities/dump/size_human'
|
13
17
|
require 'pg_export/entities/dump/base'
|
14
|
-
require 'pg_export/entities/
|
15
|
-
require 'pg_export/entities/
|
18
|
+
require 'pg_export/entities/plain_dump'
|
19
|
+
require 'pg_export/entities/encrypted_dump'
|
16
20
|
require 'pg_export/services/ftp_service'
|
17
21
|
require 'pg_export/services/ftp_service/connection'
|
18
22
|
require 'pg_export/services/utils'
|
19
23
|
require 'pg_export/services/dump_storage'
|
24
|
+
require 'pg_export/services/aes'
|
20
25
|
|
21
26
|
class PgExport
|
22
27
|
include Concurrency
|
@@ -42,14 +47,22 @@ class PgExport
|
|
42
47
|
attr_reader :config
|
43
48
|
attr_accessor :dump, :dump_storage
|
44
49
|
|
50
|
+
def create_dump
|
51
|
+
sql_dump = utils.create_dump(config.database)
|
52
|
+
enc_dump = utils.encrypt(sql_dump)
|
53
|
+
self.dump = enc_dump
|
54
|
+
end
|
55
|
+
|
45
56
|
def initialize_dump_storage
|
46
57
|
ftp_service = FtpService.new(config.ftp_params)
|
47
58
|
self.dump_storage = DumpStorage.new(ftp_service, config.database)
|
59
|
+
self
|
48
60
|
end
|
49
61
|
|
50
|
-
def
|
51
|
-
|
52
|
-
|
53
|
-
|
62
|
+
def utils
|
63
|
+
@utils ||= Utils.new(
|
64
|
+
Aes.encryptor(config.dump_encryption_key),
|
65
|
+
Aes.decryptor(config.dump_encryption_key)
|
66
|
+
)
|
54
67
|
end
|
55
68
|
end
|
@@ -3,6 +3,7 @@ class PgExport
|
|
3
3
|
DEFAULTS = {
|
4
4
|
database: nil,
|
5
5
|
keep_dumps: ENV['KEEP_DUMPS'] || 10,
|
6
|
+
dump_encryption_key: ENV['DUMP_ENCRYPTION_KEY'],
|
6
7
|
ftp_host: ENV['BACKUP_FTP_HOST'],
|
7
8
|
ftp_user: ENV['BACKUP_FTP_USER'],
|
8
9
|
ftp_password: ENV['BACKUP_FTP_PASSWORD']
|
@@ -20,6 +21,7 @@ class PgExport
|
|
20
21
|
DEFAULTS.keys.each do |field|
|
21
22
|
raise InvalidConfigurationError, "Field #{field} is required" if send(field).nil?
|
22
23
|
end
|
24
|
+
raise InvalidConfigurationError, 'Dump password is to short. It should have at least 16 characters' if dump_encryption_key.length < 16
|
23
25
|
end
|
24
26
|
|
25
27
|
def ftp_params
|
@@ -37,7 +39,7 @@ class PgExport
|
|
37
39
|
private
|
38
40
|
|
39
41
|
def print_attr(key)
|
40
|
-
if key
|
42
|
+
if %i(ftp_password dump_encryption_key).include?(key)
|
41
43
|
if send(key)
|
42
44
|
"#{key}: #{send(key)[0..2]}***\n"
|
43
45
|
else
|
@@ -6,18 +6,28 @@ class PgExport
|
|
6
6
|
|
7
7
|
CHUNK_SIZE = (2**16).freeze
|
8
8
|
|
9
|
-
def_delegators :file, :path, :read, :write, :rewind, :size, :eof?
|
9
|
+
def_delegators :file, :path, :read, :write, :<<, :rewind, :close, :size, :eof?
|
10
10
|
|
11
11
|
def initialize
|
12
12
|
@file = Tempfile.new('dump')
|
13
13
|
end
|
14
14
|
|
15
15
|
def ext
|
16
|
-
|
16
|
+
''
|
17
17
|
end
|
18
18
|
|
19
|
-
def open
|
20
|
-
|
19
|
+
def open(operation_type, &block)
|
20
|
+
case operation_type.to_sym
|
21
|
+
when :read then File.open(path, 'r', &block)
|
22
|
+
when :write then File.open(path, 'w', &block)
|
23
|
+
else raise ArgumentError, 'Operation type can be only :read or :write'
|
24
|
+
end
|
25
|
+
end
|
26
|
+
|
27
|
+
def each_chunk
|
28
|
+
open(:read) do |file|
|
29
|
+
yield file.read(CHUNK_SIZE) until file.eof?
|
30
|
+
end
|
21
31
|
end
|
22
32
|
|
23
33
|
def to_s
|
data/lib/pg_export/errors.rb
CHANGED
@@ -0,0 +1,83 @@
|
|
1
|
+
require 'pg_export/interactive/refinements/colourable_string'
|
2
|
+
|
3
|
+
class PgExport
|
4
|
+
module Interactive
|
5
|
+
include CliSpinnable
|
6
|
+
using ColourableString
|
7
|
+
|
8
|
+
def self.extended(_)
|
9
|
+
puts 'Interactive mode, for restoring dump into database.'.green
|
10
|
+
end
|
11
|
+
|
12
|
+
def call
|
13
|
+
initialize_dump_storage
|
14
|
+
print_all_dumps
|
15
|
+
download_selected_dump
|
16
|
+
restore_downloaded_dump
|
17
|
+
puts 'Success'.green
|
18
|
+
self
|
19
|
+
end
|
20
|
+
|
21
|
+
private
|
22
|
+
|
23
|
+
def initialize_dump_storage
|
24
|
+
with_spinner do |cli|
|
25
|
+
cli.print 'Connecting to FTP'
|
26
|
+
super
|
27
|
+
cli.tick
|
28
|
+
end
|
29
|
+
end
|
30
|
+
|
31
|
+
def print_all_dumps
|
32
|
+
dumps.each.with_index(0) do |name, i|
|
33
|
+
print "(#{i}) "
|
34
|
+
puts name.to_s.gray
|
35
|
+
end
|
36
|
+
self
|
37
|
+
end
|
38
|
+
|
39
|
+
def download_selected_dump
|
40
|
+
puts 'Which dump would you like to import?'
|
41
|
+
print "Type from 1 to #{dumps.count - 1} (0): "
|
42
|
+
name = dumps.fetch(gets.chomp.to_i)
|
43
|
+
with_spinner do |cli|
|
44
|
+
cli.print 'Downloading dump'
|
45
|
+
encrypted_dump = dump_storage.download(name)
|
46
|
+
cli.print " (#{encrypted_dump.size_human})"
|
47
|
+
cli.tick
|
48
|
+
cli.print 'Decrypting dump'
|
49
|
+
self.dump = utils.decrypt(encrypted_dump)
|
50
|
+
cli.print " (#{dump.size_human})"
|
51
|
+
cli.tick
|
52
|
+
end
|
53
|
+
self
|
54
|
+
rescue OpenSSL::Cipher::CipherError => e
|
55
|
+
puts "Problem decrypting dump file: #{e}. Try again.".red
|
56
|
+
retry
|
57
|
+
end
|
58
|
+
|
59
|
+
def restore_downloaded_dump
|
60
|
+
puts 'To which database you would like to restore the downloaded dump?'
|
61
|
+
if config.database == 'undefined'
|
62
|
+
print 'Enter a local database name: '
|
63
|
+
else
|
64
|
+
print "Enter a local database name (#{config.database}): "
|
65
|
+
end
|
66
|
+
database = gets.chomp
|
67
|
+
database = database.empty? ? config.database : database
|
68
|
+
with_spinner do |cli|
|
69
|
+
cli.print "Restoring dump to #{database} database"
|
70
|
+
utils.restore_dump(dump, database)
|
71
|
+
cli.tick
|
72
|
+
end
|
73
|
+
self
|
74
|
+
rescue PgRestoreError => e
|
75
|
+
puts e.to_s.red
|
76
|
+
retry
|
77
|
+
end
|
78
|
+
|
79
|
+
def dumps
|
80
|
+
@dumps ||= dump_storage.all
|
81
|
+
end
|
82
|
+
end
|
83
|
+
end
|
@@ -0,0 +1,19 @@
|
|
1
|
+
class PgExport
|
2
|
+
module Interactive
|
3
|
+
module ColourableString
|
4
|
+
refine String do
|
5
|
+
def red
|
6
|
+
"\e[31m#{self}\e[0m"
|
7
|
+
end
|
8
|
+
|
9
|
+
def green
|
10
|
+
"\e[0;32;49m#{self}\e[0m"
|
11
|
+
end
|
12
|
+
|
13
|
+
def gray
|
14
|
+
"\e[37m#{self}\e[0m"
|
15
|
+
end
|
16
|
+
end
|
17
|
+
end
|
18
|
+
end
|
19
|
+
end
|
@@ -0,0 +1,21 @@
|
|
1
|
+
class PgExport
|
2
|
+
class Aes
|
3
|
+
ALG = 'AES-128-CBC'.freeze
|
4
|
+
|
5
|
+
def self.encryptor(key)
|
6
|
+
initialize_aes(:encrypt, key)
|
7
|
+
end
|
8
|
+
|
9
|
+
def self.decryptor(key)
|
10
|
+
initialize_aes(:decrypt, key)
|
11
|
+
end
|
12
|
+
|
13
|
+
def self.initialize_aes(mode, key)
|
14
|
+
raise ArgumentError, 'Only :encrypt or :decrypt are allowed' unless %i(encrypt decrypt).include?(mode)
|
15
|
+
aes = OpenSSL::Cipher::Cipher.new(ALG)
|
16
|
+
aes.public_send(mode.to_sym)
|
17
|
+
aes.key = key
|
18
|
+
aes
|
19
|
+
end
|
20
|
+
end
|
21
|
+
end
|
@@ -12,16 +12,31 @@ class PgExport
|
|
12
12
|
def upload(dump)
|
13
13
|
dump_name = timestamped_name(dump)
|
14
14
|
ftp_service.upload_file(dump.path, dump_name)
|
15
|
-
logger.info "
|
15
|
+
logger.info "Upload #{dump} #{dump_name} to #{ftp_service}"
|
16
|
+
end
|
17
|
+
|
18
|
+
def download(name)
|
19
|
+
dump = EncryptedDump.new
|
20
|
+
ftp_service.download_file(dump.path, name)
|
21
|
+
logger.info "Download #{dump} #{name} from #{ftp_service}"
|
22
|
+
dump
|
16
23
|
end
|
17
24
|
|
18
25
|
def remove_old(keep:)
|
19
|
-
|
26
|
+
find_by_name(name).drop(keep.to_i).each do |filename|
|
20
27
|
ftp_service.delete(filename)
|
21
|
-
logger.info "Remove #{filename} from
|
28
|
+
logger.info "Remove #{filename} from #{ftp_service}"
|
22
29
|
end
|
23
30
|
end
|
24
31
|
|
32
|
+
def find_by_name(s)
|
33
|
+
ftp_service.list(s + '_*')
|
34
|
+
end
|
35
|
+
|
36
|
+
def all
|
37
|
+
ftp_service.list('*')
|
38
|
+
end
|
39
|
+
|
25
40
|
private
|
26
41
|
|
27
42
|
attr_reader :ftp_service, :name
|
@@ -29,9 +44,5 @@ class PgExport
|
|
29
44
|
def timestamped_name(dump)
|
30
45
|
name + Time.now.strftime(TIMESTAMP) + dump.ext
|
31
46
|
end
|
32
|
-
|
33
|
-
def ftp_regex
|
34
|
-
name + '_*'
|
35
|
-
end
|
36
47
|
end
|
37
48
|
end
|
@@ -9,8 +9,8 @@ class PgExport
|
|
9
9
|
ObjectSpace.define_finalizer(self, proc { connection.close })
|
10
10
|
end
|
11
11
|
|
12
|
-
def list(
|
13
|
-
ftp.list(
|
12
|
+
def list(regex_string)
|
13
|
+
ftp.list(regex_string).map { |item| item.split(' ').last }.sort.reverse
|
14
14
|
end
|
15
15
|
|
16
16
|
def delete(filename)
|
@@ -21,6 +21,10 @@ class PgExport
|
|
21
21
|
ftp.putbinaryfile(path.to_s, name, CHUNK_SIZE)
|
22
22
|
end
|
23
23
|
|
24
|
+
def download_file(path, name)
|
25
|
+
ftp.getbinaryfile(name, path.to_s, CHUNK_SIZE)
|
26
|
+
end
|
27
|
+
|
24
28
|
def to_s
|
25
29
|
host
|
26
30
|
end
|
@@ -1,37 +1,57 @@
|
|
1
1
|
class PgExport
|
2
2
|
class Utils
|
3
|
-
|
3
|
+
include Logging
|
4
4
|
|
5
|
-
def
|
6
|
-
|
7
|
-
|
8
|
-
|
5
|
+
def initialize(encryptor, decryptor)
|
6
|
+
@encryptor, @decryptor = encryptor, decryptor
|
7
|
+
end
|
8
|
+
|
9
|
+
def create_dump(database_name)
|
10
|
+
dump = PlainDump.new
|
11
|
+
Open3.popen3("pg_dump -Fc --file #{dump.path} #{database_name}") do |_, _, err|
|
12
|
+
error = err.read
|
13
|
+
raise PgDumpError, error unless error.empty?
|
14
|
+
end
|
9
15
|
logger.info "Create #{dump}"
|
10
16
|
dump
|
11
17
|
end
|
12
18
|
|
13
|
-
def
|
14
|
-
|
15
|
-
|
16
|
-
|
17
|
-
gz.write(f.read(Dump::Base::CHUNK_SIZE)) until f.eof?
|
18
|
-
end
|
19
|
+
def restore_dump(dump, database_name)
|
20
|
+
Open3.popen3("pg_restore -c -d #{database_name} #{dump.path}") do |_, _, err|
|
21
|
+
error = err.read
|
22
|
+
raise PgRestoreError, error if /FATAL/ =~ error
|
19
23
|
end
|
20
|
-
|
21
|
-
|
22
|
-
dump_gz
|
24
|
+
logger.info "Restore #{dump}"
|
25
|
+
self
|
23
26
|
end
|
24
27
|
|
25
|
-
def
|
26
|
-
|
27
|
-
|
28
|
-
|
29
|
-
|
30
|
-
|
31
|
-
end
|
28
|
+
def encrypt(dump)
|
29
|
+
enc_dump = EncryptedDump.new
|
30
|
+
copy_using(encryptor, from: dump, to: enc_dump)
|
31
|
+
logger.info "Create #{enc_dump}"
|
32
|
+
enc_dump
|
33
|
+
end
|
32
34
|
|
35
|
+
def decrypt(enc_dump)
|
36
|
+
dump = PlainDump.new
|
37
|
+
copy_using(decryptor, from: enc_dump, to: dump)
|
33
38
|
logger.info "Create #{dump}"
|
34
39
|
dump
|
35
40
|
end
|
41
|
+
|
42
|
+
private
|
43
|
+
|
44
|
+
attr_reader :encryptor, :decryptor
|
45
|
+
|
46
|
+
def copy_using(aes, from:, to:)
|
47
|
+
aes.reset
|
48
|
+
to.open(:write) do |f|
|
49
|
+
from.each_chunk do |chunk|
|
50
|
+
f << aes.update(chunk)
|
51
|
+
end
|
52
|
+
f << aes.final
|
53
|
+
end
|
54
|
+
self
|
55
|
+
end
|
36
56
|
end
|
37
57
|
end
|
data/lib/pg_export/version.rb
CHANGED
data/pg_export.gemspec
CHANGED
@@ -20,6 +20,8 @@ Gem::Specification.new do |spec|
|
|
20
20
|
spec.require_paths = ['lib']
|
21
21
|
spec.required_ruby_version = '>= 2.1.0'
|
22
22
|
|
23
|
+
spec.add_dependency 'cli_spinnable', '~> 0.2'
|
24
|
+
|
23
25
|
spec.add_development_dependency 'bundler', '~> 1.10'
|
24
26
|
spec.add_development_dependency 'rubocop', '~> 0.44'
|
25
27
|
spec.add_development_dependency 'rake', '~> 10.0'
|
metadata
CHANGED
@@ -1,15 +1,29 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: pg_export
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.4.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Krzysztof Maicher
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2016-
|
11
|
+
date: 2016-11-24 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
|
+
- !ruby/object:Gem::Dependency
|
14
|
+
name: cli_spinnable
|
15
|
+
requirement: !ruby/object:Gem::Requirement
|
16
|
+
requirements:
|
17
|
+
- - "~>"
|
18
|
+
- !ruby/object:Gem::Version
|
19
|
+
version: '0.2'
|
20
|
+
type: :runtime
|
21
|
+
prerelease: false
|
22
|
+
version_requirements: !ruby/object:Gem::Requirement
|
23
|
+
requirements:
|
24
|
+
- - "~>"
|
25
|
+
- !ruby/object:Gem::Version
|
26
|
+
version: '0.2'
|
13
27
|
- !ruby/object:Gem::Dependency
|
14
28
|
name: bundler
|
15
29
|
requirement: !ruby/object:Gem::Requirement
|
@@ -116,15 +130,19 @@ files:
|
|
116
130
|
- bin/console
|
117
131
|
- bin/pg_export
|
118
132
|
- bin/setup
|
133
|
+
- codeclimate.yml
|
119
134
|
- lib/pg_export.rb
|
120
135
|
- lib/pg_export/concurrency.rb
|
121
136
|
- lib/pg_export/configuration.rb
|
122
|
-
- lib/pg_export/entities/compressed_dump.rb
|
123
137
|
- lib/pg_export/entities/dump/base.rb
|
124
138
|
- lib/pg_export/entities/dump/size_human.rb
|
125
|
-
- lib/pg_export/entities/
|
139
|
+
- lib/pg_export/entities/encrypted_dump.rb
|
140
|
+
- lib/pg_export/entities/plain_dump.rb
|
126
141
|
- lib/pg_export/errors.rb
|
142
|
+
- lib/pg_export/interactive.rb
|
143
|
+
- lib/pg_export/interactive/refinements/colourable_string.rb
|
127
144
|
- lib/pg_export/logging.rb
|
145
|
+
- lib/pg_export/services/aes.rb
|
128
146
|
- lib/pg_export/services/dump_storage.rb
|
129
147
|
- lib/pg_export/services/ftp_service.rb
|
130
148
|
- lib/pg_export/services/ftp_service/connection.rb
|
@@ -1,19 +0,0 @@
|
|
1
|
-
class PgExport
|
2
|
-
class CompressedDump < Dump::Base
|
3
|
-
def name
|
4
|
-
'Compressed Dump'
|
5
|
-
end
|
6
|
-
|
7
|
-
def ext
|
8
|
-
'.gz'
|
9
|
-
end
|
10
|
-
|
11
|
-
def open(operation_type, &block)
|
12
|
-
case operation_type.to_sym
|
13
|
-
when :read then Zlib::GzipReader.open(path, &block)
|
14
|
-
when :write then Zlib::GzipWriter.open(path, &block)
|
15
|
-
else raise ArgumentError, 'Operation type can be only :read or :write'
|
16
|
-
end
|
17
|
-
end
|
18
|
-
end
|
19
|
-
end
|
@@ -1,19 +0,0 @@
|
|
1
|
-
class PgExport
|
2
|
-
class SqlDump < Dump::Base
|
3
|
-
def name
|
4
|
-
'Dump'
|
5
|
-
end
|
6
|
-
|
7
|
-
def ext
|
8
|
-
''
|
9
|
-
end
|
10
|
-
|
11
|
-
def open(operation_type, &block)
|
12
|
-
case operation_type.to_sym
|
13
|
-
when :read then File.open(path, 'r', &block)
|
14
|
-
when :write then File.open(path, 'w', &block)
|
15
|
-
else raise ArgumentError, 'Operation type can be only :read or :write'
|
16
|
-
end
|
17
|
-
end
|
18
|
-
end
|
19
|
-
end
|