zfs_mgmt 0.3.5 → 0.3.10
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/Gemfile +1 -0
- data/Gemfile.lock +3 -1
- data/README.md +54 -22
- data/bin/zfsmgr +11 -26
- data/lib/zfs_mgmt.rb +64 -34
- data/lib/zfs_mgmt/restic.rb +154 -25
- data/lib/zfs_mgmt/version.rb +1 -1
- data/lib/zfs_mgmt/zfs_mgr.rb +6 -0
- data/lib/zfs_mgmt/zfs_mgr/list.rb +26 -0
- data/lib/zfs_mgmt/zfs_mgr/restic.rb +41 -0
- data/lib/zfs_mgmt/zfs_mgr/snapshot.rb +27 -0
- metadata +6 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: a54ed2e0ad255f69367aaeb25fd0dddcc8346cec0fd8be02561e4ea1210510f2
|
4
|
+
data.tar.gz: 1d31c311f63835c3f7a2b6e771b06628f1901b62130e9ec5d0255ff987fb89bc
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: c0534143034e6edb6ce1161bacd86765cf9a151812d1849f3570c983691dfdf8d03f5d0aeefb070fa144fa0fa6838711b855e07180a7c70cd14f3f00d561d562
|
7
|
+
data.tar.gz: dbf53bcfe6223106f080393621d58ef538a2cf9fe9ae9e7f338b2cbe4e5d88d5bca2fcf44f3c6e8ffb76050ce2d0cfc75fa12638b8d19aae759b9b507fab3a65
|
data/Gemfile
CHANGED
data/Gemfile.lock
CHANGED
@@ -1,13 +1,14 @@
|
|
1
1
|
PATH
|
2
2
|
remote: .
|
3
3
|
specs:
|
4
|
-
zfs_mgmt (0.3.
|
4
|
+
zfs_mgmt (0.3.10)
|
5
5
|
|
6
6
|
GEM
|
7
7
|
remote: https://rubygems.org/
|
8
8
|
specs:
|
9
9
|
diff-lcs (1.3)
|
10
10
|
filesize (0.2.0)
|
11
|
+
json (2.3.0)
|
11
12
|
rake (13.0.1)
|
12
13
|
rspec (3.9.0)
|
13
14
|
rspec-core (~> 3.9.0)
|
@@ -31,6 +32,7 @@ PLATFORMS
|
|
31
32
|
DEPENDENCIES
|
32
33
|
bundler (~> 1.16)
|
33
34
|
filesize
|
35
|
+
json
|
34
36
|
rake (>= 12.3.3)
|
35
37
|
rspec (~> 3.0)
|
36
38
|
text-table
|
data/README.md
CHANGED
@@ -1,8 +1,11 @@
|
|
1
1
|
# ZfsMgmt
|
2
2
|
|
3
|
-
zfs_mgmt aims to provide some useful helpers for managing zfs
|
3
|
+
zfs_mgmt aims to provide some useful helpers for managing zfs
|
4
|
+
snapshots, and eventually send/recv duties via the zfsmgr script in
|
5
|
+
bin/.
|
4
6
|
|
5
|
-
Currently only snapshot destruction is implemented by a policy
|
7
|
+
Currently only snapshot destruction is implemented by a policy
|
8
|
+
specification stored in zfs properties.
|
6
9
|
|
7
10
|
## Installation
|
8
11
|
|
@@ -16,25 +19,36 @@ Therefore, building the gem and installing, or running ruby inside the src/ dire
|
|
16
19
|
|
17
20
|
## Usage
|
18
21
|
|
19
|
-
The most common usage pattern would be to set zfs properties as
|
22
|
+
The most common usage pattern would be to set zfs properties as
|
23
|
+
explained below, then use **zfsmgr snapshot policy** to print a table
|
24
|
+
of what would be kept and for what reason. Then use **zfsmgr snapshot destroy --noop**
|
25
|
+
to see what would be destroyed, and finally **zfsmgr snapshot destroy**
|
26
|
+
without the --noop option to actually remove snapshots.
|
20
27
|
|
28
|
+
$ zfsmgr
|
21
29
|
Commands:
|
22
30
|
zfsmgr help [COMMAND] # Describe available commands or one specific command
|
31
|
+
zfsmgr list SUBCOMMAND ...ARGS # list filesystems
|
32
|
+
zfsmgr restic SUBCOMMAND ...ARGS # backup zfs to restic
|
23
33
|
zfsmgr snapshot SUBCOMMAND ...ARGS # manage snapshots
|
24
34
|
zfsmgr zfsget [ZFS] # execute zfs get for the given properties and types and parse the output into a nested hash
|
25
|
-
|
26
|
-
|
27
|
-
|
28
|
-
|
29
|
-
|
30
|
-
|
31
|
-
|
32
|
-
|
33
|
-
|
34
|
-
|
35
|
-
|
36
|
-
|
37
|
-
|
35
|
+
|
36
|
+
### list stale
|
37
|
+
|
38
|
+
**zfsmgr list stale will** will list all zfs with "stale" snapshots,
|
39
|
+
that is the newest snapshot is older than 1d or older than the --age
|
40
|
+
parameter.
|
41
|
+
|
42
|
+
### restic backup
|
43
|
+
|
44
|
+
zfsmgr can pipe zfs send output into restic, to allow storing zfs
|
45
|
+
streams in restic repositories. zfsmgr takes an opinionated approach
|
46
|
+
to this task, implementing a traditional full/differential/incremental
|
47
|
+
backup scheme. This should allow a recent snapshot to be restored
|
48
|
+
requiring only 3 restic snapshots to recreate the filesystem. See the
|
49
|
+
properties section for restic related zfs properties.
|
50
|
+
|
51
|
+
$ zfsmgr restic backup full
|
38
52
|
|
39
53
|
|
40
54
|
## Example output
|
@@ -100,20 +114,30 @@ The most common usage pattern would be to set zfs properties as explained below,
|
|
100
114
|
|
101
115
|
## Development
|
102
116
|
|
103
|
-
After checking out the repo, run `bin/setup` to install
|
117
|
+
After checking out the repo, run `bin/setup` to install
|
118
|
+
dependencies. Then, run `rake spec` to run the tests. You can also run
|
119
|
+
`bin/console` for an interactive prompt that will allow you to
|
120
|
+
experiment.
|
104
121
|
|
105
|
-
To install this gem onto your local machine, run `bundle exec rake
|
122
|
+
To install this gem onto your local machine, run `bundle exec rake
|
123
|
+
install`. To release a new version, update the version number in
|
124
|
+
`version.rb`, and then run `bundle exec rake release`, which will
|
125
|
+
create a git tag for the version, push git commits and tags, and push
|
126
|
+
the `.gem` file to [rubygems.org](https://rubygems.org).
|
106
127
|
|
107
128
|
## Contributing
|
108
129
|
|
109
|
-
Bug reports and pull requests are welcome on GitHub at
|
130
|
+
Bug reports and pull requests are welcome on GitHub at
|
131
|
+
https://github.com/aranc23/zfs_mgmt.
|
110
132
|
|
111
133
|
## zfs user properties
|
112
134
|
|
113
|
-
Destruction of zfs snapshots is based on the following zfs user
|
135
|
+
Destruction of zfs snapshots is based on the following zfs user
|
136
|
+
properties:
|
114
137
|
|
115
138
|
### zfsmgmt:manage
|
116
|
-
manage snapshots for this filesystem if this property is 'true'
|
139
|
+
manage snapshots for this filesystem if this property is 'true'
|
140
|
+
(string literal)
|
117
141
|
|
118
142
|
### zfsmgmt:policy
|
119
143
|
|
@@ -127,7 +151,8 @@ Examples:
|
|
127
151
|
- 1y1m1y1d1h (1 of each time frame worth of snapshots)
|
128
152
|
- 72h (72 hourly snapshots)
|
129
153
|
|
130
|
-
The order in which each timeframe is listed in does not matter, and
|
154
|
+
The order in which each timeframe is listed in does not matter, and
|
155
|
+
the supported specs are as follows:
|
131
156
|
|
132
157
|
- h - hourly
|
133
158
|
- d - daily
|
@@ -194,6 +219,13 @@ create snapshots called autosnap-%FT%T%z.
|
|
194
219
|
strftime format string used when creating snapshot names, default
|
195
220
|
being %FT%T%z.
|
196
221
|
|
222
|
+
### zfsmgmt:restic_backup
|
223
|
+
boolean, send this zfs to restic
|
224
|
+
|
225
|
+
### zfsmgmt:restic_repository
|
226
|
+
send the zfs to this repository, optional, rely on restic environment
|
227
|
+
variables otherwise
|
228
|
+
|
197
229
|
## Snapshot Management / zfs destroy
|
198
230
|
When destroying snapshots according to a given policy, all snapshots
|
199
231
|
should be considered for deletion and all snapshots should be
|
data/bin/zfsmgr
CHANGED
@@ -1,44 +1,29 @@
|
|
1
1
|
require "thor"
|
2
2
|
require "zfs_mgmt"
|
3
|
-
|
4
|
-
|
5
|
-
|
6
|
-
:desc => 'pass -n option to zfs commands'
|
7
|
-
class_option :verbose, :type => :boolean, :default => false,
|
8
|
-
:desc => 'pass -v option to zfs commands'
|
9
|
-
class_option :debug, :type => :boolean, :default => false,
|
10
|
-
:desc => 'set logging level to debug'
|
11
|
-
class_option :filter, :type => :string, :default => '.+',
|
12
|
-
:desc => 'only act on zfs matching this regexp'
|
13
|
-
desc "destroy", "apply the snapshot destroy policy to zfs"
|
14
|
-
def destroy()
|
15
|
-
ZfsMgmt.snapshot_destroy(noop: options[:noop], verbopt: options[:verbose], debugopt: options[:debug], filter: options[:filter])
|
16
|
-
end
|
17
|
-
desc "policy", "print the policy table for zfs"
|
18
|
-
def policy()
|
19
|
-
ZfsMgmt.snapshot_policy(verbopt: options[:verbose], debugopt: options[:debug], filter: options[:filter])
|
20
|
-
end
|
21
|
-
desc "create", "execute zfs snapshot based on zfs properties"
|
22
|
-
def create()
|
23
|
-
ZfsMgmt.snapshot_create(verbopt: options[:verbose], debugopt: options[:debug], filter: options[:filter])
|
24
|
-
end
|
25
|
-
end
|
3
|
+
require "zfs_mgmt/restic"
|
4
|
+
require "zfs_mgmt/zfs_mgr"
|
5
|
+
require "zfs_mgmt/zfs_mgr/restic"
|
26
6
|
|
27
7
|
class ZfsMgr < Thor
|
8
|
+
class_option :zfs_binary, :type => :string, :default => 'zfs',
|
9
|
+
:desc => 'zfs binary'
|
10
|
+
class_option :zpool_binary, :type => :string, :default => 'zpool',
|
11
|
+
:desc => 'zfs binary'
|
28
12
|
desc "zfsget [ZFS]", "execute zfs get for the given properties and types and parse the output into a nested hash"
|
29
13
|
method_option :properties, :type => :array, :default => ['name'], :desc => "List of properties passed to zfs get"
|
30
14
|
method_option :types, :type => :array, :default => ['filesystem','volume'], enum: ['filesystem','volume','snapshot'], :desc => "list of types"
|
31
15
|
def zfsget(zfs)
|
16
|
+
ZfsMgmt.global_options = options
|
32
17
|
pp ZfsMgmt.zfsget(properties: options[:properties],
|
33
18
|
types: options[:types],
|
34
19
|
zfs: zfs)
|
35
20
|
end
|
36
21
|
desc "snapshot SUBCOMMAND ...ARGS", "manage snapshots"
|
37
|
-
subcommand "snapshot", Snapshot
|
22
|
+
subcommand "snapshot", ZfsMgmt::ZfsMgr::Snapshot
|
38
23
|
desc "list SUBCOMMAND ...ARGS", "list filesystems"
|
39
|
-
subcommand "list",
|
24
|
+
subcommand "list", ZfsMgmt::ZfsMgr::List
|
40
25
|
desc "restic SUBCOMMAND ...ARGS", "backup zfs to restic"
|
41
|
-
subcommand "restic",
|
26
|
+
subcommand "restic", ZfsMgmt::ZfsMgr::Restic
|
42
27
|
end
|
43
28
|
|
44
29
|
ZfsMgr.start(ARGV)
|
data/lib/zfs_mgmt.rb
CHANGED
@@ -1,6 +1,7 @@
|
|
1
1
|
# coding: utf-8
|
2
2
|
require "zfs_mgmt/version"
|
3
3
|
require "zfs_mgmt/restic"
|
4
|
+
require "zfs_mgmt/zfs_mgr"
|
4
5
|
require 'pp'
|
5
6
|
require 'date'
|
6
7
|
require 'logger'
|
@@ -37,6 +38,9 @@ $properties_xlate = {
|
|
37
38
|
}
|
38
39
|
|
39
40
|
module ZfsMgmt
|
41
|
+
class << self
|
42
|
+
attr_accessor :global_options
|
43
|
+
end
|
40
44
|
def self.custom_properties()
|
41
45
|
return [
|
42
46
|
'policy',
|
@@ -64,10 +68,49 @@ module ZfsMgmt
|
|
64
68
|
return md[1].to_i
|
65
69
|
end
|
66
70
|
end
|
67
|
-
|
71
|
+
|
72
|
+
def self.zfs_holds(snapshot)
|
73
|
+
com = [global_options['zfs_binary'], 'holds', '-H', snapshot]
|
74
|
+
$logger.debug("#{com.join(' ')}")
|
75
|
+
out = %x(#{com.join(' ')})
|
76
|
+
unless $?.success?
|
77
|
+
errstr = "unable to retrieves holds for snapshot: #{snapshot}"
|
78
|
+
$logger.error(errstr)
|
79
|
+
raise errstr
|
80
|
+
end
|
81
|
+
a = []
|
82
|
+
out.split("\n").each do |ln|
|
83
|
+
a.push(ln.split("\t")[1])
|
84
|
+
end
|
85
|
+
a
|
86
|
+
end
|
87
|
+
|
88
|
+
def self.zfs_hold(hold,snapshot)
|
89
|
+
com = [global_options['zfs_binary'], 'hold', hold, snapshot]
|
90
|
+
$logger.debug("#{com.join(' ')}")
|
91
|
+
system(com.join(' '))
|
92
|
+
unless $?.success?
|
93
|
+
errstr = "unable to set hold: #{hold} for snapshot: #{snapshot}"
|
94
|
+
$logger.error(errstr)
|
95
|
+
raise errstr
|
96
|
+
end
|
97
|
+
end
|
98
|
+
|
99
|
+
def self.zfs_release(hold,snapshot)
|
100
|
+
com = [@global_options['zfs_binary'], 'release', hold, snapshot]
|
101
|
+
$logger.debug("#{com.join(' ')}")
|
102
|
+
system(com.join(' '))
|
103
|
+
unless $?.success?
|
104
|
+
errstr = "unable to release hold: #{hold} for snapshot: #{snapshot}"
|
105
|
+
$logger.error(errstr)
|
106
|
+
raise errstr
|
107
|
+
end
|
108
|
+
end
|
109
|
+
|
68
110
|
def self.zfsget(properties: ['name'],types: ['filesystem','volume'],zfs: '')
|
69
111
|
results={}
|
70
|
-
com = [
|
112
|
+
com = [ZfsMgmt.global_options[:zfs_binary], 'get', '-Hp', properties.join(','), '-t', types.join(','), zfs]
|
113
|
+
$logger.debug(com.join(' '))
|
71
114
|
so,se,status = Open3.capture3(com.join(' '))
|
72
115
|
if status.signaled?
|
73
116
|
$logger.error("process was signalled \"#{com.join(' ')}\", termsig #{status.termsig}")
|
@@ -192,24 +235,25 @@ module ZfsMgmt
|
|
192
235
|
}
|
193
236
|
return saved,saved_snaps,deleteme
|
194
237
|
end
|
195
|
-
def self.zfs_managed_list(filter: '.+')
|
238
|
+
def self.zfs_managed_list(filter: '.+', properties: custom_properties(), property_match: { 'zfsmgmt:manage' => 'true' } )
|
196
239
|
zfss = [] # array of arrays
|
197
|
-
zfsget(properties:
|
240
|
+
zfsget(properties: properties).each do |zfs,props|
|
198
241
|
unless /#{filter}/ =~ zfs
|
199
242
|
next
|
200
243
|
end
|
201
|
-
|
202
|
-
|
244
|
+
managed = true
|
245
|
+
property_match.each do |k,v|
|
246
|
+
unless props.has_key?(k) and props[k] == v
|
247
|
+
managed = false
|
248
|
+
break
|
249
|
+
end
|
203
250
|
end
|
251
|
+
next unless managed
|
204
252
|
snaps = self.zfsget(properties: ['name','creation','userrefs','used','written','referenced'],types: ['snapshot'], zfs: zfs)
|
205
253
|
if snaps.length == 0
|
206
254
|
$logger.warn("unable to process this zfs, no snapshots at all: #{zfs}")
|
207
255
|
next
|
208
256
|
end
|
209
|
-
unless props.has_key?('zfsmgmt:policy') and policy = policy_parser(props['zfsmgmt:policy'])
|
210
|
-
$logger.error("zfs_mgmt is configured to manage #{zfs}, but there is no valid policy configuration, skipping")
|
211
|
-
next # zfs
|
212
|
-
end
|
213
257
|
zfss.push([zfs,props,snaps])
|
214
258
|
end
|
215
259
|
return zfss
|
@@ -222,6 +266,10 @@ module ZfsMgmt
|
|
222
266
|
end
|
223
267
|
zfs_managed_list(filter: filter).each do |zdata|
|
224
268
|
(zfs,props,snaps) = zdata
|
269
|
+
unless props.has_key?('zfsmgmt:policy') and policy_parser(props['zfsmgmt:policy'])
|
270
|
+
$logger.error("zfs_mgmt is configured to manage #{zfs}, but there is no valid policy configuration, skipping")
|
271
|
+
next # zfs
|
272
|
+
end
|
225
273
|
# call the function that decides who to save and who to delete
|
226
274
|
(saved,saved_snaps,deleteme) = snapshot_destroy_policy(zfs,props,snaps)
|
227
275
|
|
@@ -247,6 +295,11 @@ module ZfsMgmt
|
|
247
295
|
end
|
248
296
|
zfs_managed_list(filter: filter).each do |zdata|
|
249
297
|
(zfs,props,snaps) = zdata
|
298
|
+
unless props.has_key?('zfsmgmt:policy') and policy_parser(props['zfsmgmt:policy'])
|
299
|
+
$logger.error("zfs_mgmt is configured to manage #{zfs}, but there is no valid policy configuration, skipping")
|
300
|
+
next # zfs
|
301
|
+
end
|
302
|
+
|
250
303
|
# call the function that decides who to save and who to delete
|
251
304
|
(saved,saved_snaps,deleteme) = snapshot_destroy_policy(zfs,props,snaps)
|
252
305
|
|
@@ -322,7 +375,7 @@ module ZfsMgmt
|
|
322
375
|
|
323
376
|
prefix = ( props.has_key?('zfsmgmt:snap_prefix') ? props['zfsmgmt:snap_prefix'] : 'zfsmgmt' )
|
324
377
|
ts = ( props.has_key?('zfsmgmt:snap_timestamp') ? props['zfsmgmt:snap_timestamp'] : '%FT%T%z' )
|
325
|
-
com = ['
|
378
|
+
com = [global_options['zfs_binary'],'snapshot']
|
326
379
|
if props['zfsmgmt:snapshot'] == 'recursive' and props['zfsmgmt:snapshot@source'] == 'local'
|
327
380
|
com.push('-r')
|
328
381
|
end
|
@@ -333,26 +386,3 @@ module ZfsMgmt
|
|
333
386
|
end
|
334
387
|
end
|
335
388
|
end
|
336
|
-
class ZfsMgmtList < Thor
|
337
|
-
class_option :filter, :type => :string, :default => '.+',
|
338
|
-
:desc => 'only act on zfs matching this regexp'
|
339
|
-
desc "stale", "list all zfs with stale snapshots"
|
340
|
-
method_option :age, :desc => "timeframe outside of which the zfs will be considered stale", :default => '1d'
|
341
|
-
def stale()
|
342
|
-
cutoff = Time.at(Time.now.to_i - ZfsMgmt.timespec_to_seconds(options[:age]))
|
343
|
-
table = Text::Table.new
|
344
|
-
table.head = ['zfs','snapshot','age']
|
345
|
-
table.rows = []
|
346
|
-
ZfsMgmt.zfs_managed_list(filter: options[:filter]).each do |blob|
|
347
|
-
zfs,props,snaps = blob
|
348
|
-
last = snaps.keys.sort { |a,b| snaps[a]['creation'] <=> snaps[b]['creation'] }.last
|
349
|
-
snap_time = Time.at(snaps[last]['creation'])
|
350
|
-
if snap_time < cutoff
|
351
|
-
table.rows << [zfs,last.split('@')[1],snap_time]
|
352
|
-
end
|
353
|
-
end
|
354
|
-
if table.rows.count > 0
|
355
|
-
print table.to_s
|
356
|
-
end
|
357
|
-
end
|
358
|
-
end
|
data/lib/zfs_mgmt/restic.rb
CHANGED
@@ -1,33 +1,162 @@
|
|
1
|
+
require "json"
|
1
2
|
|
2
3
|
module ZfsMgmt::Restic
|
3
|
-
def self.
|
4
|
-
|
5
|
-
|
6
|
-
|
4
|
+
def self.restic_snapshots(zfs,options,props)
|
5
|
+
# query the restic database
|
6
|
+
com = [ options[:restic_binary],
|
7
|
+
'snapshots',
|
8
|
+
'--json',
|
9
|
+
'--tag', 'zfsmgmt',
|
10
|
+
'--path', "/#{zfs}",
|
11
|
+
]
|
12
|
+
if options.has_key?('password_file')
|
13
|
+
com.push('-p',options['password_file'])
|
14
|
+
end
|
15
|
+
if options.has_key?('repo')
|
16
|
+
com.push('--repo', options['repo'])
|
17
|
+
elsif props.has_key?('zfsmgmt:restic_repository')
|
18
|
+
com.push( '--repo', props['zfsmgmt:restic_repository'] )
|
19
|
+
end
|
7
20
|
|
8
|
-
|
9
|
-
|
10
|
-
|
11
|
-
|
12
|
-
|
13
|
-
|
14
|
-
|
15
|
-
|
16
|
-
|
17
|
-
|
18
|
-
|
19
|
-
|
20
|
-
|
21
|
-
|
22
|
-
|
23
|
-
|
21
|
+
$logger.debug("#{com.join(' ')}")
|
22
|
+
restic_output = %x(#{com.join(' ')})
|
23
|
+
unless $?.success?
|
24
|
+
$logger.error("unable to query the restic database")
|
25
|
+
raise "unable to query the restic database"
|
26
|
+
end
|
27
|
+
restic_snapshots = JSON.parse(restic_output)
|
28
|
+
restic_snapshot_zfs_snapshot_index = {}
|
29
|
+
restic_snapshots.each do |snappy|
|
30
|
+
snappy['date_time'] = DateTime.parse(snappy['time'])
|
31
|
+
if snappy.has_key?('tags')
|
32
|
+
snappy['tags'].each do |t|
|
33
|
+
if m = /^(zfsmgmt:.+?)=(.+)/.match(t)
|
34
|
+
if ['zfsmgmt:level'].include?(m[1])
|
35
|
+
snappy[m[1]] = m[2].to_i
|
36
|
+
else
|
37
|
+
snappy[m[1]] = m[2]
|
38
|
+
end
|
39
|
+
if m[1] == 'zfsmgmt:snapshot'
|
40
|
+
restic_snapshot_zfs_snapshot_index[m[2]] = snappy
|
41
|
+
end
|
42
|
+
end
|
43
|
+
end
|
44
|
+
end
|
45
|
+
end
|
46
|
+
return([restic_snapshots,restic_snapshot_zfs_snapshot_index])
|
24
47
|
end
|
25
|
-
end
|
26
48
|
|
27
|
-
|
28
|
-
|
29
|
-
|
49
|
+
def self.valid_chain(snap,restic_snapshots,restic_snapshot_zfs_snapshot_index,a)
|
50
|
+
if snap['zfsmgmt:level'] == 0
|
51
|
+
a.push(snap)
|
52
|
+
$logger.debug("found complete chain culminating in full backup of: #{snap['zfsmgmt:snapshot']}")
|
53
|
+
return a
|
54
|
+
elsif restic_snapshot_zfs_snapshot_index.has_key?(snap['zfsmgmt:parent'])
|
55
|
+
a.push(snap)
|
56
|
+
$logger.debug("found another link in the chain: #{snap['zfsmgmt:snapshot']} => #{snap['zfsmgmt:parent']}")
|
57
|
+
return valid_chain(restic_snapshot_zfs_snapshot_index[snap['zfsmgmt:parent']],restic_snapshots,restic_snapshot_zfs_snapshot_index,a)
|
58
|
+
else
|
59
|
+
$logger.error("broken chain: looking for the parent of #{snap['zfsmgmt:snapshot']} (#{snap['zfsmgmt:parent']}) and failed to find")
|
60
|
+
return []
|
61
|
+
end
|
62
|
+
end
|
63
|
+
|
64
|
+
|
65
|
+
def self.backup(backup_level: 2,
|
66
|
+
options: {})
|
67
|
+
ZfsMgmt.zfs_managed_list(filter: options['filter'],
|
68
|
+
properties: ['name',
|
69
|
+
'zfsmgmt:restic_backup',
|
70
|
+
'zfsmgmt:restic_repository',
|
71
|
+
'userrefs',
|
72
|
+
],
|
73
|
+
property_match: { 'zfsmgmt:restic_backup' => 'true' }).each do |blob|
|
74
|
+
zfs,props,zfs_snapshots = blob
|
75
|
+
last_zfs_snapshot = zfs_snapshots.keys.sort { |a,b| zfs_snapshots[a]['creation'] <=> zfs_snapshots[b]['creation'] }.last
|
76
|
+
zfs_snap_time = Time.at(zfs_snapshots[last_zfs_snapshot]['creation'])
|
30
77
|
|
78
|
+
level = 0
|
79
|
+
chain = []
|
80
|
+
zfs_snap_parent = ''
|
81
|
+
restic_snap_parent = ''
|
82
|
+
(restic_snapshots,restic_snapshot_zfs_snapshot_index) = restic_snapshots(zfs,options,props)
|
83
|
+
if restic_snapshot_zfs_snapshot_index.has_key?(last_zfs_snapshot)
|
84
|
+
$logger.warn("backup of this snapshot #{last_zfs_snapshot} already exists in restic, cannot continue with backup of #{zfs}")
|
85
|
+
next # next zfs filesystem to be backed up
|
86
|
+
end
|
87
|
+
if backup_level > 0 and restic_snapshots.count > 0
|
88
|
+
# reverse (oldest first) sorted restic snapshots
|
89
|
+
restic_snap_parent = restic_snapshots.filter { |rsnap|
|
90
|
+
rsnap.has_key?('zfsmgmt:zfs') and rsnap['zfsmgmt:zfs'] == zfs and
|
91
|
+
rsnap.has_key?('zfsmgmt:level') and rsnap['zfsmgmt:level'] < backup_level }.sort {
|
92
|
+
|a,b| a['date_time'] <=> b['date_time'] }.last
|
93
|
+
if restic_snap_parent and
|
94
|
+
zfs_snapshots.has_key?(restic_snap_parent['zfsmgmt:snapshot']) and
|
95
|
+
chain = valid_chain(restic_snap_parent,restic_snapshots,restic_snapshot_zfs_snapshot_index,[]) and
|
96
|
+
chain.length > 0
|
97
|
+
|
98
|
+
level = restic_snap_parent['zfsmgmt:level'] + 1
|
99
|
+
zfs_snap_parent = restic_snap_parent['zfsmgmt:snapshot']
|
100
|
+
$logger.debug("restic_snap_parent: level: #{restic_snap_parent['zfsmgmt:level']} snapshot: #{zfs_snap_parent}")
|
101
|
+
else
|
102
|
+
$logger.error("restic_snap_parent rejected: level: #{restic_snap_parent['zfsmgmt:level']} snapshot: #{restic_snap_parent['zfsmgmt:snapshot']}")
|
103
|
+
end
|
104
|
+
$logger.debug("chain of snapshots: #{chain}")
|
105
|
+
end
|
106
|
+
tags = [ 'zfsmgmt',
|
107
|
+
"zfsmgmt:snapshot=#{last_zfs_snapshot}",
|
108
|
+
"zfsmgmt:zfs=#{zfs}",
|
109
|
+
"zfsmgmt:level=#{level}" ]
|
110
|
+
com = [ ZfsMgmt.global_options['zfs_binary'], 'send', '-L', '-w', '-h', '-p' ]
|
111
|
+
if level > 0
|
112
|
+
if options[:intermediary]
|
113
|
+
com.push('-I')
|
114
|
+
else
|
115
|
+
com.push('-i')
|
116
|
+
end
|
117
|
+
com.push(zfs_snap_parent)
|
118
|
+
tags.push("zfsmgmt:parent=#{zfs_snap_parent}")
|
119
|
+
end
|
120
|
+
com.push( last_zfs_snapshot )
|
121
|
+
com.push( '|', 'mbuffer', '-m', options[:buffer], '-q' )
|
122
|
+
com.push( '|', options[:restic_binary], 'backup', '--stdin',
|
123
|
+
'--stdin-filename', zfs, '--time', "\"#{zfs_snap_time.strftime('%F %T')}\"" )
|
124
|
+
tags.each do |tag|
|
125
|
+
com.push( '--tag', "\"#{tag}\"" )
|
126
|
+
end
|
127
|
+
if options.has_key?('limit_upload')
|
128
|
+
com.push('--limit-upload', options['limit_upload'])
|
129
|
+
end
|
130
|
+
if options.has_key?('password_file')
|
131
|
+
com.push('-p',options['password_file'])
|
132
|
+
end
|
133
|
+
if options.has_key?('repo')
|
134
|
+
com.push('--repo', options['repo'])
|
135
|
+
elsif props.has_key?('zfsmgmt:restic_repository')
|
136
|
+
com.push( '--repo', props['zfsmgmt:restic_repository'] )
|
137
|
+
end
|
138
|
+
if options[:verbose]
|
139
|
+
com.push('--verbose',options[:verbose])
|
140
|
+
elsif $stdout.isatty
|
141
|
+
com.push('-v')
|
142
|
+
end
|
143
|
+
unless ZfsMgmt.zfs_holds(last_zfs_snapshot).include?('zfsmgmt_restic')
|
144
|
+
ZfsMgmt.zfs_hold('zfsmgmt_restic',last_zfs_snapshot)
|
145
|
+
end
|
146
|
+
$logger.info("#{com.join(' ')}")
|
147
|
+
system(com.join(' '))
|
148
|
+
chain_snaps = chain.map do |rsnap|
|
149
|
+
rsnap['zfsmgmt:snapshot']
|
150
|
+
end
|
151
|
+
zfs_snapshots.each do |s,d|
|
152
|
+
d['userrefs'] == 0 and next
|
153
|
+
chain_snaps.include?(s) and next
|
154
|
+
s == last_zfs_snapshot and next
|
155
|
+
if ZfsMgmt.zfs_holds(s).include?('zfsmgmt_restic')
|
156
|
+
ZfsMgmt.zfs_release('zfsmgmt_restic',s)
|
157
|
+
end
|
158
|
+
end
|
159
|
+
end
|
160
|
+
end
|
31
161
|
end
|
32
162
|
|
33
|
-
|
data/lib/zfs_mgmt/version.rb
CHANGED
@@ -0,0 +1,26 @@
|
|
1
|
+
# class list
|
2
|
+
|
3
|
+
class ZfsMgmt::ZfsMgr::List < Thor
|
4
|
+
class_option :filter, :type => :string, :default => '.+',
|
5
|
+
:desc => 'only act on zfs matching this regexp'
|
6
|
+
desc "stale", "list all zfs with stale snapshots"
|
7
|
+
method_option :age, :desc => "timeframe outside of which the zfs will be considered stale", :default => '1d'
|
8
|
+
def stale()
|
9
|
+
ZfsMgmt.global_options = options
|
10
|
+
cutoff = Time.at(Time.now.to_i - ZfsMgmt.timespec_to_seconds(options[:age]))
|
11
|
+
table = Text::Table.new
|
12
|
+
table.head = ['zfs','snapshot','age']
|
13
|
+
table.rows = []
|
14
|
+
ZfsMgmt.zfs_managed_list(filter: options[:filter]).each do |blob|
|
15
|
+
zfs,props,snaps = blob
|
16
|
+
last = snaps.keys.sort { |a,b| snaps[a]['creation'] <=> snaps[b]['creation'] }.last
|
17
|
+
snap_time = Time.at(snaps[last]['creation'])
|
18
|
+
if snap_time < cutoff
|
19
|
+
table.rows << [zfs,last.split('@')[1],snap_time]
|
20
|
+
end
|
21
|
+
end
|
22
|
+
if table.rows.count > 0
|
23
|
+
print table.to_s
|
24
|
+
end
|
25
|
+
end
|
26
|
+
end
|
@@ -0,0 +1,41 @@
|
|
1
|
+
|
2
|
+
class ZfsMgmt::ZfsMgr::Backup < Thor
|
3
|
+
include ZfsMgmt::Restic
|
4
|
+
class_option :filter, :type => :string, :default => '.+',
|
5
|
+
:desc => 'only act on zfs matching this regexp'
|
6
|
+
class_option :restic_binary, :type => :string, :default => 'restic',
|
7
|
+
:desc => 'restic binary'
|
8
|
+
class_option :verbose, :alias => '-v', :type => :numeric,
|
9
|
+
:desc => 'verbosity level for restic'
|
10
|
+
class_option :buffer, :type => :string, :default => '256m',
|
11
|
+
:desc => 'buffer size for mbuffer'
|
12
|
+
class_option :password_file, :alias => '-p', :type => :string,
|
13
|
+
:desc => 'passed to restic'
|
14
|
+
class_option :limit_upload, :type => :numeric,
|
15
|
+
:desc => 'passed to restic'
|
16
|
+
class_option :repo, :type => :string,
|
17
|
+
:desc => 'passed to restic'
|
18
|
+
desc "incremental", "perform incremental backup"
|
19
|
+
method_option :level, :desc => "backup level in integer form", :default => 2, :type => :numeric
|
20
|
+
method_option :intermediary, :alias => '-I', :desc => "pass -I (intermediary) option to zfs send", :default => false, :type => :boolean
|
21
|
+
def incremental()
|
22
|
+
ZfsMgmt.global_options = options
|
23
|
+
ZfsMgmt::Restic.backup(backup_level: options[:level], options: options)
|
24
|
+
end
|
25
|
+
desc "differential", "perform differential backup"
|
26
|
+
method_option :intermediary, :alias => '-I', :desc => "pass -I (intermediary) option to zfs send", :default => false, :type => :boolean
|
27
|
+
def differential()
|
28
|
+
ZfsMgmt.global_options = options
|
29
|
+
ZfsMgmt::Restic.backup(backup_level: 1, options: options)
|
30
|
+
end
|
31
|
+
desc "full", "perform full backup"
|
32
|
+
def full()
|
33
|
+
ZfsMgmt.global_options = options
|
34
|
+
ZfsMgmt::Restic.backup(backup_level: 0, options: options)
|
35
|
+
end
|
36
|
+
end
|
37
|
+
|
38
|
+
class ZfsMgmt::ZfsMgr::Restic < Thor
|
39
|
+
desc "backup SUBCOMMAND ...ARGS", "backup all configured zfs to restic"
|
40
|
+
subcommand "backup", ZfsMgmt::ZfsMgr::Backup
|
41
|
+
end
|
@@ -0,0 +1,27 @@
|
|
1
|
+
# implement snapshot management
|
2
|
+
|
3
|
+
class ZfsMgmt::ZfsMgr::Snapshot < Thor
|
4
|
+
class_option :noop, :type => :boolean, :default => false,
|
5
|
+
:desc => 'pass -n option to zfs commands'
|
6
|
+
class_option :verbose, :type => :boolean, :default => false,
|
7
|
+
:desc => 'pass -v option to zfs commands'
|
8
|
+
class_option :debug, :type => :boolean, :default => false,
|
9
|
+
:desc => 'set logging level to debug'
|
10
|
+
class_option :filter, :type => :string, :default => '.+',
|
11
|
+
:desc => 'only act on zfs matching this regexp'
|
12
|
+
desc "destroy", "apply the snapshot destroy policy to zfs"
|
13
|
+
def destroy()
|
14
|
+
ZfsMgmt.global_options = options
|
15
|
+
ZfsMgmt.snapshot_destroy(noop: options[:noop], verbopt: options[:verbose], debugopt: options[:debug], filter: options[:filter])
|
16
|
+
end
|
17
|
+
desc "policy", "print the policy table for zfs"
|
18
|
+
def policy()
|
19
|
+
ZfsMgmt.global_options = options
|
20
|
+
ZfsMgmt.snapshot_policy(verbopt: options[:verbose], debugopt: options[:debug], filter: options[:filter])
|
21
|
+
end
|
22
|
+
desc "create", "execute zfs snapshot based on zfs properties"
|
23
|
+
def create()
|
24
|
+
ZfsMgmt.global_options = options
|
25
|
+
ZfsMgmt.snapshot_create(verbopt: options[:verbose], debugopt: options[:debug], filter: options[:filter])
|
26
|
+
end
|
27
|
+
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: zfs_mgmt
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.3.
|
4
|
+
version: 0.3.10
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Aran Cox
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2021-02-
|
11
|
+
date: 2021-02-21 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: bundler
|
@@ -127,6 +127,10 @@ files:
|
|
127
127
|
- lib/zfs_mgmt.rb
|
128
128
|
- lib/zfs_mgmt/restic.rb
|
129
129
|
- lib/zfs_mgmt/version.rb
|
130
|
+
- lib/zfs_mgmt/zfs_mgr.rb
|
131
|
+
- lib/zfs_mgmt/zfs_mgr/list.rb
|
132
|
+
- lib/zfs_mgmt/zfs_mgr/restic.rb
|
133
|
+
- lib/zfs_mgmt/zfs_mgr/snapshot.rb
|
130
134
|
- zfs_mgmt.gemspec
|
131
135
|
homepage: https://github.com/aranc23/zfs_mgmt
|
132
136
|
licenses:
|