martilla 0.2.9 → 0.3.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.gitignore +4 -0
- data/.travis.yml +4 -1
- data/Gemfile.lock +5 -5
- data/README.md +45 -33
- data/lib/martilla/backup.rb +5 -3
- data/lib/martilla/databases/mysql.rb +1 -1
- data/lib/martilla/notifier.rb +12 -2
- data/lib/martilla/notifiers/ses.rb +1 -3
- data/lib/martilla/notifiers/slack.rb +3 -2
- data/lib/martilla/storage.rb +12 -0
- data/lib/martilla/storages/local.rb +29 -0
- data/lib/martilla/storages/s3.rb +10 -0
- data/lib/martilla/storages/scp.rb +4 -0
- data/lib/martilla/version.rb +1 -1
- metadata +2 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: aaf89471f6777ba461ddf9769030ef39b30a019c76e10dc35236cf8bcca03a53
|
4
|
+
data.tar.gz: 192c63a35a2e7d0381c45643d86defe699bcdaadc750ccbf8c782aeadf52fbbb
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 596ccdd75c799206a00f363ccd9c203eb93671c70296127efb197087d621e2209fa584f4ef73b1f5fe2fa9ff5d895807e926bf32764e752803eb02eadff320b3
|
7
|
+
data.tar.gz: dbada619ce14429929942cd842396d5aae9551bcc177e2438b48dabdc79d5623adb63d73db752e23fef0f575579412f5eac4014c0e20a26f153f02b4d8f66efc
|
data/.gitignore
CHANGED
data/.travis.yml
CHANGED
data/Gemfile.lock
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
PATH
|
2
2
|
remote: .
|
3
3
|
specs:
|
4
|
-
martilla (0.
|
4
|
+
martilla (0.3.0)
|
5
5
|
aws-sdk-s3 (~> 1.49)
|
6
6
|
aws-sdk-ses (~> 1.26)
|
7
7
|
memoist (~> 0.16.0)
|
@@ -13,8 +13,8 @@ GEM
|
|
13
13
|
remote: https://rubygems.org/
|
14
14
|
specs:
|
15
15
|
aws-eventstream (1.0.3)
|
16
|
-
aws-partitions (1.
|
17
|
-
aws-sdk-core (3.
|
16
|
+
aws-partitions (1.237.0)
|
17
|
+
aws-sdk-core (3.76.0)
|
18
18
|
aws-eventstream (~> 1.0, >= 1.0.2)
|
19
19
|
aws-partitions (~> 1, >= 1.228.0)
|
20
20
|
aws-sigv4 (~> 1.1)
|
@@ -22,7 +22,7 @@ GEM
|
|
22
22
|
aws-sdk-kms (1.25.0)
|
23
23
|
aws-sdk-core (~> 3, >= 3.71.0)
|
24
24
|
aws-sigv4 (~> 1.1)
|
25
|
-
aws-sdk-s3 (1.
|
25
|
+
aws-sdk-s3 (1.53.0)
|
26
26
|
aws-sdk-core (~> 3, >= 3.71.0)
|
27
27
|
aws-sdk-kms (~> 1)
|
28
28
|
aws-sigv4 (~> 1.1)
|
@@ -39,7 +39,7 @@ GEM
|
|
39
39
|
json (2.2.0)
|
40
40
|
mail (2.7.1)
|
41
41
|
mini_mime (>= 0.1.1)
|
42
|
-
memoist (0.16.
|
42
|
+
memoist (0.16.1)
|
43
43
|
method_source (0.9.2)
|
44
44
|
mini_mime (1.0.2)
|
45
45
|
pony (1.13.1)
|
data/README.md
CHANGED
@@ -72,10 +72,12 @@ Currently available DB types to choose from are **postgres** & **mysql**. They b
|
|
72
72
|
|
73
73
|
### Storages
|
74
74
|
|
75
|
-
The available Storages types are
|
75
|
+
The available Storages types are **local**, **S3**& **SCP**. They each have different available options:
|
76
76
|
- options for type: **local**
|
77
77
|
- `filename`
|
78
78
|
- The location to where the backup will be stored
|
79
|
+
- `retention`
|
80
|
+
- An integer that defines the max number of backups stored at the defined location
|
79
81
|
- options for type: **s3**
|
80
82
|
- `filename`
|
81
83
|
- The location to where the backup will be stored within the S3 bucket
|
@@ -85,51 +87,61 @@ The available Storages types are 'local', 'S3' & 'SCP'. They each have different
|
|
85
87
|
- can be specified with the usual ENV variables or IAM roles
|
86
88
|
- `secret_access_key`
|
87
89
|
- can be specified with the usual ENV variables or IAM roles
|
90
|
+
- `retention`
|
91
|
+
- An integer that defines the max number of backups stored at the defined location
|
88
92
|
- options for type: **scp**
|
89
93
|
- `filename`
|
90
94
|
- The location to where the backup will be stored within remote server
|
91
95
|
- `host`
|
92
96
|
- `user`
|
93
97
|
- `identity_file`
|
98
|
+
- `retention`
|
99
|
+
- Not implemented for this storage ([see #12](https://github.com/fdoxyz/martilla/issues/12))
|
94
100
|
|
95
101
|
All storage types also accept 'suffix' as a boolean that enables or disables a timestamp to be added as a suffix to the backup 'filename', it defaults as `true`.
|
96
102
|
|
97
103
|
### Notifiers
|
98
104
|
|
99
|
-
The available Notifiers are
|
100
|
-
- options for type: **ses** (email notifier)
|
101
|
-
|
102
|
-
|
103
|
-
|
104
|
-
|
105
|
-
|
106
|
-
- options for type: **sendmail** (email notifier)
|
107
|
-
|
108
|
-
- options for type: **smtp** (email notifier)
|
109
|
-
|
110
|
-
|
111
|
-
|
112
|
-
|
113
|
-
- options for type: **slack**
|
114
|
-
|
115
|
-
|
116
|
-
|
117
|
-
|
118
|
-
|
119
|
-
|
120
|
-
|
121
|
-
|
122
|
-
|
105
|
+
The available Notifiers are **ses**, **sendmail** & **smtp**. They each have different available options:
|
106
|
+
- options for type: **ses** (email notifier)
|
107
|
+
- `aws_region`
|
108
|
+
- `access_key_id`
|
109
|
+
- can be specified with the usual ENV variables or IAM role
|
110
|
+
- `secret_access_key`
|
111
|
+
- can be specified with the usual ENV variables or IAM roles
|
112
|
+
- options for type: **sendmail** (email notifier)
|
113
|
+
- no custom options
|
114
|
+
- options for type: **smtp** (email notifier)
|
115
|
+
- `address`
|
116
|
+
- `domain`
|
117
|
+
- `user_name`
|
118
|
+
- `password`
|
119
|
+
- options for type: **slack**
|
120
|
+
- `slack_webhook_url`
|
121
|
+
- required
|
122
|
+
- more info [here](https://api.slack.com/messaging/webhooks)
|
123
|
+
- `slack_channel`
|
124
|
+
- defaults to `#general`
|
125
|
+
- the channel to post the backup notifications
|
126
|
+
- `slack_username`
|
127
|
+
- defaults to `Martilla`
|
128
|
+
- the username which backup notifications will be posted with
|
123
129
|
|
124
130
|
All of the previous **email notifiers** also have the following options that can be customized:
|
125
|
-
|
126
|
-
|
127
|
-
|
128
|
-
|
129
|
-
|
130
|
-
|
131
|
-
|
132
|
-
|
131
|
+
- `to`
|
132
|
+
- a list of comma separated emails to be notified
|
133
|
+
- `from`
|
134
|
+
- defaults to 'martilla@no-reply.com'
|
135
|
+
- `success_subject`
|
136
|
+
- the subject of the success email
|
137
|
+
- `failure_subject`
|
138
|
+
- the subject of the failure email
|
139
|
+
|
140
|
+
Also **ALL** notifiers have the following two options
|
141
|
+
- `send_success`
|
142
|
+
- `Boolean` value that will disable notifications on success when set to false. Defaults to `true`
|
143
|
+
- `send_failure`
|
144
|
+
- `Boolean` value that will disable notifications on failure when set to false. Defaults to `true`
|
133
145
|
|
134
146
|
It's **HIGHLY RECOMMENDED** to test and make sure emails are being delivered correctly to each target inbox. Emails with standard messages like these automated backup notifications tend to be easily marked as spam.
|
135
147
|
|
data/lib/martilla/backup.rb
CHANGED
@@ -45,14 +45,15 @@ module Martilla
|
|
45
45
|
|
46
46
|
# Persist the backup
|
47
47
|
@storage.persist(tmp_file: tmp_file, gzip: gzip?)
|
48
|
+
@storage.enfore_retention!(gzip: gzip?)
|
48
49
|
rescue Exception => e
|
49
50
|
@notifiers.each do |notifier|
|
50
|
-
notifier.error(e.message, metadata)
|
51
|
+
notifier.error(e.message, metadata) if notifier.send_failure?
|
51
52
|
end
|
52
53
|
puts "An error occurred: #{e.inspect}"
|
53
54
|
else
|
54
55
|
@notifiers.each do |notifier|
|
55
|
-
notifier.success(metadata)
|
56
|
+
notifier.success(metadata) if notifier.send_success?
|
56
57
|
end
|
57
58
|
puts "Backup created and persisted successfully"
|
58
59
|
end
|
@@ -101,7 +102,8 @@ module Martilla
|
|
101
102
|
'storage' => {
|
102
103
|
'type' => 'local',
|
103
104
|
'options' => {
|
104
|
-
'filename' => 'database-backup.sql'
|
105
|
+
'filename' => 'database-backup.sql',
|
106
|
+
'retention' => 0
|
105
107
|
}
|
106
108
|
},
|
107
109
|
'notifiers' => [
|
data/lib/martilla/notifier.rb
CHANGED
@@ -1,5 +1,3 @@
|
|
1
|
-
require 'forwardable'
|
2
|
-
|
3
1
|
module Martilla
|
4
2
|
class Notifier < Component
|
5
3
|
attr_reader :options
|
@@ -21,6 +19,18 @@ module Martilla
|
|
21
19
|
'Notifier configuration is invalid. Details here: https://github.com/fdoxyz/martilla'
|
22
20
|
end
|
23
21
|
|
22
|
+
def send_success?
|
23
|
+
value = @options['send_success']
|
24
|
+
return true if value.nil?
|
25
|
+
value
|
26
|
+
end
|
27
|
+
|
28
|
+
def send_failure?
|
29
|
+
value = @options['send_failure']
|
30
|
+
return true if value.nil?
|
31
|
+
value
|
32
|
+
end
|
33
|
+
|
24
34
|
# When a new Notifier is supported it needs to go here
|
25
35
|
def self.create(config = {})
|
26
36
|
case config['type'].downcase
|
@@ -25,8 +25,9 @@ module Martilla
|
|
25
25
|
end
|
26
26
|
|
27
27
|
def slack_webhook_url
|
28
|
-
|
29
|
-
|
28
|
+
webhook_url = @options['slack_webhook_url'] || ENV['SLACK_WEBHOOK_URL']
|
29
|
+
raise config_error('slack_webhook_url') if webhook_url.nil?
|
30
|
+
webhook_url
|
30
31
|
end
|
31
32
|
|
32
33
|
def slack_channel
|
data/lib/martilla/storage.rb
CHANGED
@@ -13,6 +13,10 @@ module Martilla
|
|
13
13
|
raise NotImplementedError, 'You must implement the persist method'
|
14
14
|
end
|
15
15
|
|
16
|
+
def enforce_retention!
|
17
|
+
raise NotImplementedError, 'You must implement the enforce_retention! method'
|
18
|
+
end
|
19
|
+
|
16
20
|
def invalid_options_msg
|
17
21
|
'Storage configuration is invalid. Details here: https://github.com/fdoxyz/martilla'
|
18
22
|
end
|
@@ -42,6 +46,14 @@ module Martilla
|
|
42
46
|
"#{dirname}/#{basename}_#{timestamp}#{extension}"
|
43
47
|
end
|
44
48
|
|
49
|
+
def retention_limit
|
50
|
+
@options['retention'].to_i
|
51
|
+
end
|
52
|
+
|
53
|
+
def timestamp_regex
|
54
|
+
/\d{4}-\d{2}-\d{2}T\d{6}/
|
55
|
+
end
|
56
|
+
|
45
57
|
def config_error(config_name)
|
46
58
|
Error.new("Storage adapter configuration requires #{config_name}. Details here: https://github.com/fdoxyz/martilla")
|
47
59
|
end
|
@@ -5,5 +5,34 @@ module Martilla
|
|
5
5
|
return nil if $?.success?
|
6
6
|
raise Error.new("Local storage failed with code #{$?.exitstatus}")
|
7
7
|
end
|
8
|
+
|
9
|
+
def enfore_retention!(gzip:)
|
10
|
+
return if retention_limit < 1
|
11
|
+
files = backup_file_list(output_filename(gzip))
|
12
|
+
|
13
|
+
while files.count > retention_limit do
|
14
|
+
File.delete(files.first)
|
15
|
+
puts "Retention limit met. Removed the backup file: #{files.shift}"
|
16
|
+
end
|
17
|
+
end
|
18
|
+
|
19
|
+
private
|
20
|
+
|
21
|
+
# Oldest first & most recent last
|
22
|
+
def backup_file_list(sample_filename)
|
23
|
+
dirname = File.dirname(sample_filename)
|
24
|
+
basename = File.basename(sample_filename, '.*')
|
25
|
+
|
26
|
+
if suffix?
|
27
|
+
# Replaces file's basename timestamp for wildcards to match againts
|
28
|
+
# what exist in the directory. Ex: "backup_2019-11-10T114342" will be
|
29
|
+
# replaced with "backup_*-*-*T*". This means all backups will match
|
30
|
+
# using `Dir.glob` below
|
31
|
+
sections = basename.split('_').reject { |str| timestamp_regex =~ str }
|
32
|
+
basename = "#{sections.join('_')}_*-*-*T*"
|
33
|
+
end
|
34
|
+
|
35
|
+
Dir["#{dirname}/#{basename}.*"].sort_by { |f| File.mtime(f) }
|
36
|
+
end
|
8
37
|
end
|
9
38
|
end
|
data/lib/martilla/storages/s3.rb
CHANGED
@@ -11,6 +11,16 @@ module Martilla
|
|
11
11
|
raise Error.new('Error uploading backup to bucket')
|
12
12
|
end
|
13
13
|
|
14
|
+
def enfore_retention!(gzip:)
|
15
|
+
return if retention_limit < 1
|
16
|
+
|
17
|
+
objs = s3_resource.bucket(bucket_name).objects.sort_by(&:last_modified)
|
18
|
+
while objs.count > retention_limit do
|
19
|
+
objs.first.delete
|
20
|
+
puts "Retention limit met. Removed the backup file: #{objs.shift.key}"
|
21
|
+
end
|
22
|
+
end
|
23
|
+
|
14
24
|
private
|
15
25
|
|
16
26
|
def s3_resource
|
@@ -6,6 +6,10 @@ module Martilla
|
|
6
6
|
raise Error.new("SCP storage failed with code #{$?.exitstatus}")
|
7
7
|
end
|
8
8
|
|
9
|
+
def enfore_retention!(gzip:)
|
10
|
+
puts 'WARNING: Retention is not implemented for SCP storage. More details: https://github.com/fdoxyz/martilla/issues/12'
|
11
|
+
end
|
12
|
+
|
9
13
|
private
|
10
14
|
|
11
15
|
def host
|
data/lib/martilla/version.rb
CHANGED
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: martilla
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.3.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Fernando Valverde
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2019-
|
11
|
+
date: 2019-11-11 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: thor
|