zfs_mgmt 0.3.5 → 0.3.10
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/Gemfile +1 -0
- data/Gemfile.lock +3 -1
- data/README.md +54 -22
- data/bin/zfsmgr +11 -26
- data/lib/zfs_mgmt.rb +64 -34
- data/lib/zfs_mgmt/restic.rb +154 -25
- data/lib/zfs_mgmt/version.rb +1 -1
- data/lib/zfs_mgmt/zfs_mgr.rb +6 -0
- data/lib/zfs_mgmt/zfs_mgr/list.rb +26 -0
- data/lib/zfs_mgmt/zfs_mgr/restic.rb +41 -0
- data/lib/zfs_mgmt/zfs_mgr/snapshot.rb +27 -0
- metadata +6 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: a54ed2e0ad255f69367aaeb25fd0dddcc8346cec0fd8be02561e4ea1210510f2
|
4
|
+
data.tar.gz: 1d31c311f63835c3f7a2b6e771b06628f1901b62130e9ec5d0255ff987fb89bc
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: c0534143034e6edb6ce1161bacd86765cf9a151812d1849f3570c983691dfdf8d03f5d0aeefb070fa144fa0fa6838711b855e07180a7c70cd14f3f00d561d562
|
7
|
+
data.tar.gz: dbf53bcfe6223106f080393621d58ef538a2cf9fe9ae9e7f338b2cbe4e5d88d5bca2fcf44f3c6e8ffb76050ce2d0cfc75fa12638b8d19aae759b9b507fab3a65
|
data/Gemfile
CHANGED
data/Gemfile.lock
CHANGED
@@ -1,13 +1,14 @@
|
|
1
1
|
PATH
|
2
2
|
remote: .
|
3
3
|
specs:
|
4
|
-
zfs_mgmt (0.3.
|
4
|
+
zfs_mgmt (0.3.10)
|
5
5
|
|
6
6
|
GEM
|
7
7
|
remote: https://rubygems.org/
|
8
8
|
specs:
|
9
9
|
diff-lcs (1.3)
|
10
10
|
filesize (0.2.0)
|
11
|
+
json (2.3.0)
|
11
12
|
rake (13.0.1)
|
12
13
|
rspec (3.9.0)
|
13
14
|
rspec-core (~> 3.9.0)
|
@@ -31,6 +32,7 @@ PLATFORMS
|
|
31
32
|
DEPENDENCIES
|
32
33
|
bundler (~> 1.16)
|
33
34
|
filesize
|
35
|
+
json
|
34
36
|
rake (>= 12.3.3)
|
35
37
|
rspec (~> 3.0)
|
36
38
|
text-table
|
data/README.md
CHANGED
@@ -1,8 +1,11 @@
|
|
1
1
|
# ZfsMgmt
|
2
2
|
|
3
|
-
zfs_mgmt aims to provide some useful helpers for managing zfs
|
3
|
+
zfs_mgmt aims to provide some useful helpers for managing zfs
|
4
|
+
snapshots, and eventually send/recv duties via the zfsmgr script in
|
5
|
+
bin/.
|
4
6
|
|
5
|
-
Currently only snapshot destruction is implemented by a policy
|
7
|
+
Currently only snapshot destruction is implemented by a policy
|
8
|
+
specification stored in zfs properties.
|
6
9
|
|
7
10
|
## Installation
|
8
11
|
|
@@ -16,25 +19,36 @@ Therefore, building the gem and installing, or running ruby inside the src/ dire
|
|
16
19
|
|
17
20
|
## Usage
|
18
21
|
|
19
|
-
The most common usage pattern would be to set zfs properties as
|
22
|
+
The most common usage pattern would be to set zfs properties as
|
23
|
+
explained below, then use **zfsmgr snapshot policy** to print a table
|
24
|
+
of what would be kept and for what reason. Then use **zfsmgr snapshot destroy --noop**
|
25
|
+
to see what would be destroyed, and finally **zfsmgr snapshot destroy**
|
26
|
+
without the --noop option to actually remove snapshots.
|
20
27
|
|
28
|
+
$ zfsmgr
|
21
29
|
Commands:
|
22
30
|
zfsmgr help [COMMAND] # Describe available commands or one specific command
|
31
|
+
zfsmgr list SUBCOMMAND ...ARGS # list filesystems
|
32
|
+
zfsmgr restic SUBCOMMAND ...ARGS # backup zfs to restic
|
23
33
|
zfsmgr snapshot SUBCOMMAND ...ARGS # manage snapshots
|
24
34
|
zfsmgr zfsget [ZFS] # execute zfs get for the given properties and types and parse the output into a nested hash
|
25
|
-
|
26
|
-
|
27
|
-
|
28
|
-
|
29
|
-
|
30
|
-
|
31
|
-
|
32
|
-
|
33
|
-
|
34
|
-
|
35
|
-
|
36
|
-
|
37
|
-
|
35
|
+
|
36
|
+
### list stale
|
37
|
+
|
38
|
+
**zfsmgr list stale will** will list all zfs with "stale" snapshots,
|
39
|
+
that is the newest snapshot is older than 1d or older than the --age
|
40
|
+
parameter.
|
41
|
+
|
42
|
+
### restic backup
|
43
|
+
|
44
|
+
zfsmgr can pipe zfs send output into restic, to allow storing zfs
|
45
|
+
streams in restic repositories. zfsmgr takes an opinionated approach
|
46
|
+
to this task, implementing a traditional full/differential/incremental
|
47
|
+
backup scheme. This should allow a recent snapshot to be restored
|
48
|
+
requiring only 3 restic snapshots to recreate the filesystem. See the
|
49
|
+
properties section for restic related zfs properties.
|
50
|
+
|
51
|
+
$ zfsmgr restic backup full
|
38
52
|
|
39
53
|
|
40
54
|
## Example output
|
@@ -100,20 +114,30 @@ The most common usage pattern would be to set zfs properties as explained below,
|
|
100
114
|
|
101
115
|
## Development
|
102
116
|
|
103
|
-
After checking out the repo, run `bin/setup` to install
|
117
|
+
After checking out the repo, run `bin/setup` to install
|
118
|
+
dependencies. Then, run `rake spec` to run the tests. You can also run
|
119
|
+
`bin/console` for an interactive prompt that will allow you to
|
120
|
+
experiment.
|
104
121
|
|
105
|
-
To install this gem onto your local machine, run `bundle exec rake
|
122
|
+
To install this gem onto your local machine, run `bundle exec rake
|
123
|
+
install`. To release a new version, update the version number in
|
124
|
+
`version.rb`, and then run `bundle exec rake release`, which will
|
125
|
+
create a git tag for the version, push git commits and tags, and push
|
126
|
+
the `.gem` file to [rubygems.org](https://rubygems.org).
|
106
127
|
|
107
128
|
## Contributing
|
108
129
|
|
109
|
-
Bug reports and pull requests are welcome on GitHub at
|
130
|
+
Bug reports and pull requests are welcome on GitHub at
|
131
|
+
https://github.com/aranc23/zfs_mgmt.
|
110
132
|
|
111
133
|
## zfs user properties
|
112
134
|
|
113
|
-
Destruction of zfs snapshots is based on the following zfs user
|
135
|
+
Destruction of zfs snapshots is based on the following zfs user
|
136
|
+
properties:
|
114
137
|
|
115
138
|
### zfsmgmt:manage
|
116
|
-
manage snapshots for this filesystem if this property is 'true'
|
139
|
+
manage snapshots for this filesystem if this property is 'true'
|
140
|
+
(string literal)
|
117
141
|
|
118
142
|
### zfsmgmt:policy
|
119
143
|
|
@@ -127,7 +151,8 @@ Examples:
|
|
127
151
|
- 1y1m1y1d1h (1 of each time frame worth of snapshots)
|
128
152
|
- 72h (72 hourly snapshots)
|
129
153
|
|
130
|
-
The order in which each timeframe is listed in does not matter, and
|
154
|
+
The order in which each timeframe is listed in does not matter, and
|
155
|
+
the supported specs are as follows:
|
131
156
|
|
132
157
|
- h - hourly
|
133
158
|
- d - daily
|
@@ -194,6 +219,13 @@ create snapshots called autosnap-%FT%T%z.
|
|
194
219
|
strftime format string used when creating snapshot names, default
|
195
220
|
being %FT%T%z.
|
196
221
|
|
222
|
+
### zfsmgmt:restic_backup
|
223
|
+
boolean, send this zfs to restic
|
224
|
+
|
225
|
+
### zfsmgmt:restic_repository
|
226
|
+
send the zfs to this repository, optional, rely on restic environment
|
227
|
+
variables otherwise
|
228
|
+
|
197
229
|
## Snapshot Management / zfs destroy
|
198
230
|
When destroying snapshots according to a given policy, all snapshots
|
199
231
|
should be considered for deletion and all snapshots should be
|
data/bin/zfsmgr
CHANGED
@@ -1,44 +1,29 @@
|
|
1
1
|
require "thor"
|
2
2
|
require "zfs_mgmt"
|
3
|
-
|
4
|
-
|
5
|
-
|
6
|
-
:desc => 'pass -n option to zfs commands'
|
7
|
-
class_option :verbose, :type => :boolean, :default => false,
|
8
|
-
:desc => 'pass -v option to zfs commands'
|
9
|
-
class_option :debug, :type => :boolean, :default => false,
|
10
|
-
:desc => 'set logging level to debug'
|
11
|
-
class_option :filter, :type => :string, :default => '.+',
|
12
|
-
:desc => 'only act on zfs matching this regexp'
|
13
|
-
desc "destroy", "apply the snapshot destroy policy to zfs"
|
14
|
-
def destroy()
|
15
|
-
ZfsMgmt.snapshot_destroy(noop: options[:noop], verbopt: options[:verbose], debugopt: options[:debug], filter: options[:filter])
|
16
|
-
end
|
17
|
-
desc "policy", "print the policy table for zfs"
|
18
|
-
def policy()
|
19
|
-
ZfsMgmt.snapshot_policy(verbopt: options[:verbose], debugopt: options[:debug], filter: options[:filter])
|
20
|
-
end
|
21
|
-
desc "create", "execute zfs snapshot based on zfs properties"
|
22
|
-
def create()
|
23
|
-
ZfsMgmt.snapshot_create(verbopt: options[:verbose], debugopt: options[:debug], filter: options[:filter])
|
24
|
-
end
|
25
|
-
end
|
3
|
+
require "zfs_mgmt/restic"
|
4
|
+
require "zfs_mgmt/zfs_mgr"
|
5
|
+
require "zfs_mgmt/zfs_mgr/restic"
|
26
6
|
|
27
7
|
class ZfsMgr < Thor
|
8
|
+
class_option :zfs_binary, :type => :string, :default => 'zfs',
|
9
|
+
:desc => 'zfs binary'
|
10
|
+
class_option :zpool_binary, :type => :string, :default => 'zpool',
|
11
|
+
:desc => 'zfs binary'
|
28
12
|
desc "zfsget [ZFS]", "execute zfs get for the given properties and types and parse the output into a nested hash"
|
29
13
|
method_option :properties, :type => :array, :default => ['name'], :desc => "List of properties passed to zfs get"
|
30
14
|
method_option :types, :type => :array, :default => ['filesystem','volume'], enum: ['filesystem','volume','snapshot'], :desc => "list of types"
|
31
15
|
def zfsget(zfs)
|
16
|
+
ZfsMgmt.global_options = options
|
32
17
|
pp ZfsMgmt.zfsget(properties: options[:properties],
|
33
18
|
types: options[:types],
|
34
19
|
zfs: zfs)
|
35
20
|
end
|
36
21
|
desc "snapshot SUBCOMMAND ...ARGS", "manage snapshots"
|
37
|
-
subcommand "snapshot", Snapshot
|
22
|
+
subcommand "snapshot", ZfsMgmt::ZfsMgr::Snapshot
|
38
23
|
desc "list SUBCOMMAND ...ARGS", "list filesystems"
|
39
|
-
subcommand "list",
|
24
|
+
subcommand "list", ZfsMgmt::ZfsMgr::List
|
40
25
|
desc "restic SUBCOMMAND ...ARGS", "backup zfs to restic"
|
41
|
-
subcommand "restic",
|
26
|
+
subcommand "restic", ZfsMgmt::ZfsMgr::Restic
|
42
27
|
end
|
43
28
|
|
44
29
|
ZfsMgr.start(ARGV)
|
data/lib/zfs_mgmt.rb
CHANGED
@@ -1,6 +1,7 @@
|
|
1
1
|
# coding: utf-8
|
2
2
|
require "zfs_mgmt/version"
|
3
3
|
require "zfs_mgmt/restic"
|
4
|
+
require "zfs_mgmt/zfs_mgr"
|
4
5
|
require 'pp'
|
5
6
|
require 'date'
|
6
7
|
require 'logger'
|
@@ -37,6 +38,9 @@ $properties_xlate = {
|
|
37
38
|
}
|
38
39
|
|
39
40
|
module ZfsMgmt
|
41
|
+
class << self
|
42
|
+
attr_accessor :global_options
|
43
|
+
end
|
40
44
|
def self.custom_properties()
|
41
45
|
return [
|
42
46
|
'policy',
|
@@ -64,10 +68,49 @@ module ZfsMgmt
|
|
64
68
|
return md[1].to_i
|
65
69
|
end
|
66
70
|
end
|
67
|
-
|
71
|
+
|
72
|
+
def self.zfs_holds(snapshot)
|
73
|
+
com = [global_options['zfs_binary'], 'holds', '-H', snapshot]
|
74
|
+
$logger.debug("#{com.join(' ')}")
|
75
|
+
out = %x(#{com.join(' ')})
|
76
|
+
unless $?.success?
|
77
|
+
errstr = "unable to retrieves holds for snapshot: #{snapshot}"
|
78
|
+
$logger.error(errstr)
|
79
|
+
raise errstr
|
80
|
+
end
|
81
|
+
a = []
|
82
|
+
out.split("\n").each do |ln|
|
83
|
+
a.push(ln.split("\t")[1])
|
84
|
+
end
|
85
|
+
a
|
86
|
+
end
|
87
|
+
|
88
|
+
def self.zfs_hold(hold,snapshot)
|
89
|
+
com = [global_options['zfs_binary'], 'hold', hold, snapshot]
|
90
|
+
$logger.debug("#{com.join(' ')}")
|
91
|
+
system(com.join(' '))
|
92
|
+
unless $?.success?
|
93
|
+
errstr = "unable to set hold: #{hold} for snapshot: #{snapshot}"
|
94
|
+
$logger.error(errstr)
|
95
|
+
raise errstr
|
96
|
+
end
|
97
|
+
end
|
98
|
+
|
99
|
+
def self.zfs_release(hold,snapshot)
|
100
|
+
com = [@global_options['zfs_binary'], 'release', hold, snapshot]
|
101
|
+
$logger.debug("#{com.join(' ')}")
|
102
|
+
system(com.join(' '))
|
103
|
+
unless $?.success?
|
104
|
+
errstr = "unable to release hold: #{hold} for snapshot: #{snapshot}"
|
105
|
+
$logger.error(errstr)
|
106
|
+
raise errstr
|
107
|
+
end
|
108
|
+
end
|
109
|
+
|
68
110
|
def self.zfsget(properties: ['name'],types: ['filesystem','volume'],zfs: '')
|
69
111
|
results={}
|
70
|
-
com = [
|
112
|
+
com = [ZfsMgmt.global_options[:zfs_binary], 'get', '-Hp', properties.join(','), '-t', types.join(','), zfs]
|
113
|
+
$logger.debug(com.join(' '))
|
71
114
|
so,se,status = Open3.capture3(com.join(' '))
|
72
115
|
if status.signaled?
|
73
116
|
$logger.error("process was signalled \"#{com.join(' ')}\", termsig #{status.termsig}")
|
@@ -192,24 +235,25 @@ module ZfsMgmt
|
|
192
235
|
}
|
193
236
|
return saved,saved_snaps,deleteme
|
194
237
|
end
|
195
|
-
def self.zfs_managed_list(filter: '.+')
|
238
|
+
def self.zfs_managed_list(filter: '.+', properties: custom_properties(), property_match: { 'zfsmgmt:manage' => 'true' } )
|
196
239
|
zfss = [] # array of arrays
|
197
|
-
zfsget(properties:
|
240
|
+
zfsget(properties: properties).each do |zfs,props|
|
198
241
|
unless /#{filter}/ =~ zfs
|
199
242
|
next
|
200
243
|
end
|
201
|
-
|
202
|
-
|
244
|
+
managed = true
|
245
|
+
property_match.each do |k,v|
|
246
|
+
unless props.has_key?(k) and props[k] == v
|
247
|
+
managed = false
|
248
|
+
break
|
249
|
+
end
|
203
250
|
end
|
251
|
+
next unless managed
|
204
252
|
snaps = self.zfsget(properties: ['name','creation','userrefs','used','written','referenced'],types: ['snapshot'], zfs: zfs)
|
205
253
|
if snaps.length == 0
|
206
254
|
$logger.warn("unable to process this zfs, no snapshots at all: #{zfs}")
|
207
255
|
next
|
208
256
|
end
|
209
|
-
unless props.has_key?('zfsmgmt:policy') and policy = policy_parser(props['zfsmgmt:policy'])
|
210
|
-
$logger.error("zfs_mgmt is configured to manage #{zfs}, but there is no valid policy configuration, skipping")
|
211
|
-
next # zfs
|
212
|
-
end
|
213
257
|
zfss.push([zfs,props,snaps])
|
214
258
|
end
|
215
259
|
return zfss
|
@@ -222,6 +266,10 @@ module ZfsMgmt
|
|
222
266
|
end
|
223
267
|
zfs_managed_list(filter: filter).each do |zdata|
|
224
268
|
(zfs,props,snaps) = zdata
|
269
|
+
unless props.has_key?('zfsmgmt:policy') and policy_parser(props['zfsmgmt:policy'])
|
270
|
+
$logger.error("zfs_mgmt is configured to manage #{zfs}, but there is no valid policy configuration, skipping")
|
271
|
+
next # zfs
|
272
|
+
end
|
225
273
|
# call the function that decides who to save and who to delete
|
226
274
|
(saved,saved_snaps,deleteme) = snapshot_destroy_policy(zfs,props,snaps)
|
227
275
|
|
@@ -247,6 +295,11 @@ module ZfsMgmt
|
|
247
295
|
end
|
248
296
|
zfs_managed_list(filter: filter).each do |zdata|
|
249
297
|
(zfs,props,snaps) = zdata
|
298
|
+
unless props.has_key?('zfsmgmt:policy') and policy_parser(props['zfsmgmt:policy'])
|
299
|
+
$logger.error("zfs_mgmt is configured to manage #{zfs}, but there is no valid policy configuration, skipping")
|
300
|
+
next # zfs
|
301
|
+
end
|
302
|
+
|
250
303
|
# call the function that decides who to save and who to delete
|
251
304
|
(saved,saved_snaps,deleteme) = snapshot_destroy_policy(zfs,props,snaps)
|
252
305
|
|
@@ -322,7 +375,7 @@ module ZfsMgmt
|
|
322
375
|
|
323
376
|
prefix = ( props.has_key?('zfsmgmt:snap_prefix') ? props['zfsmgmt:snap_prefix'] : 'zfsmgmt' )
|
324
377
|
ts = ( props.has_key?('zfsmgmt:snap_timestamp') ? props['zfsmgmt:snap_timestamp'] : '%FT%T%z' )
|
325
|
-
com = ['
|
378
|
+
com = [global_options['zfs_binary'],'snapshot']
|
326
379
|
if props['zfsmgmt:snapshot'] == 'recursive' and props['zfsmgmt:snapshot@source'] == 'local'
|
327
380
|
com.push('-r')
|
328
381
|
end
|
@@ -333,26 +386,3 @@ module ZfsMgmt
|
|
333
386
|
end
|
334
387
|
end
|
335
388
|
end
|
336
|
-
class ZfsMgmtList < Thor
|
337
|
-
class_option :filter, :type => :string, :default => '.+',
|
338
|
-
:desc => 'only act on zfs matching this regexp'
|
339
|
-
desc "stale", "list all zfs with stale snapshots"
|
340
|
-
method_option :age, :desc => "timeframe outside of which the zfs will be considered stale", :default => '1d'
|
341
|
-
def stale()
|
342
|
-
cutoff = Time.at(Time.now.to_i - ZfsMgmt.timespec_to_seconds(options[:age]))
|
343
|
-
table = Text::Table.new
|
344
|
-
table.head = ['zfs','snapshot','age']
|
345
|
-
table.rows = []
|
346
|
-
ZfsMgmt.zfs_managed_list(filter: options[:filter]).each do |blob|
|
347
|
-
zfs,props,snaps = blob
|
348
|
-
last = snaps.keys.sort { |a,b| snaps[a]['creation'] <=> snaps[b]['creation'] }.last
|
349
|
-
snap_time = Time.at(snaps[last]['creation'])
|
350
|
-
if snap_time < cutoff
|
351
|
-
table.rows << [zfs,last.split('@')[1],snap_time]
|
352
|
-
end
|
353
|
-
end
|
354
|
-
if table.rows.count > 0
|
355
|
-
print table.to_s
|
356
|
-
end
|
357
|
-
end
|
358
|
-
end
|
data/lib/zfs_mgmt/restic.rb
CHANGED
@@ -1,33 +1,162 @@
|
|
1
|
+
require "json"
|
1
2
|
|
2
3
|
module ZfsMgmt::Restic
|
3
|
-
def self.
|
4
|
-
|
5
|
-
|
6
|
-
|
4
|
+
def self.restic_snapshots(zfs,options,props)
|
5
|
+
# query the restic database
|
6
|
+
com = [ options[:restic_binary],
|
7
|
+
'snapshots',
|
8
|
+
'--json',
|
9
|
+
'--tag', 'zfsmgmt',
|
10
|
+
'--path', "/#{zfs}",
|
11
|
+
]
|
12
|
+
if options.has_key?('password_file')
|
13
|
+
com.push('-p',options['password_file'])
|
14
|
+
end
|
15
|
+
if options.has_key?('repo')
|
16
|
+
com.push('--repo', options['repo'])
|
17
|
+
elsif props.has_key?('zfsmgmt:restic_repository')
|
18
|
+
com.push( '--repo', props['zfsmgmt:restic_repository'] )
|
19
|
+
end
|
7
20
|
|
8
|
-
|
9
|
-
|
10
|
-
|
11
|
-
|
12
|
-
|
13
|
-
|
14
|
-
|
15
|
-
|
16
|
-
|
17
|
-
|
18
|
-
|
19
|
-
|
20
|
-
|
21
|
-
|
22
|
-
|
23
|
-
|
21
|
+
$logger.debug("#{com.join(' ')}")
|
22
|
+
restic_output = %x(#{com.join(' ')})
|
23
|
+
unless $?.success?
|
24
|
+
$logger.error("unable to query the restic database")
|
25
|
+
raise "unable to query the restic database"
|
26
|
+
end
|
27
|
+
restic_snapshots = JSON.parse(restic_output)
|
28
|
+
restic_snapshot_zfs_snapshot_index = {}
|
29
|
+
restic_snapshots.each do |snappy|
|
30
|
+
snappy['date_time'] = DateTime.parse(snappy['time'])
|
31
|
+
if snappy.has_key?('tags')
|
32
|
+
snappy['tags'].each do |t|
|
33
|
+
if m = /^(zfsmgmt:.+?)=(.+)/.match(t)
|
34
|
+
if ['zfsmgmt:level'].include?(m[1])
|
35
|
+
snappy[m[1]] = m[2].to_i
|
36
|
+
else
|
37
|
+
snappy[m[1]] = m[2]
|
38
|
+
end
|
39
|
+
if m[1] == 'zfsmgmt:snapshot'
|
40
|
+
restic_snapshot_zfs_snapshot_index[m[2]] = snappy
|
41
|
+
end
|
42
|
+
end
|
43
|
+
end
|
44
|
+
end
|
45
|
+
end
|
46
|
+
return([restic_snapshots,restic_snapshot_zfs_snapshot_index])
|
24
47
|
end
|
25
|
-
end
|
26
48
|
|
27
|
-
|
28
|
-
|
29
|
-
|
49
|
+
def self.valid_chain(snap,restic_snapshots,restic_snapshot_zfs_snapshot_index,a)
|
50
|
+
if snap['zfsmgmt:level'] == 0
|
51
|
+
a.push(snap)
|
52
|
+
$logger.debug("found complete chain culminating in full backup of: #{snap['zfsmgmt:snapshot']}")
|
53
|
+
return a
|
54
|
+
elsif restic_snapshot_zfs_snapshot_index.has_key?(snap['zfsmgmt:parent'])
|
55
|
+
a.push(snap)
|
56
|
+
$logger.debug("found another link in the chain: #{snap['zfsmgmt:snapshot']} => #{snap['zfsmgmt:parent']}")
|
57
|
+
return valid_chain(restic_snapshot_zfs_snapshot_index[snap['zfsmgmt:parent']],restic_snapshots,restic_snapshot_zfs_snapshot_index,a)
|
58
|
+
else
|
59
|
+
$logger.error("broken chain: looking for the parent of #{snap['zfsmgmt:snapshot']} (#{snap['zfsmgmt:parent']}) and failed to find")
|
60
|
+
return []
|
61
|
+
end
|
62
|
+
end
|
63
|
+
|
64
|
+
|
65
|
+
def self.backup(backup_level: 2,
|
66
|
+
options: {})
|
67
|
+
ZfsMgmt.zfs_managed_list(filter: options['filter'],
|
68
|
+
properties: ['name',
|
69
|
+
'zfsmgmt:restic_backup',
|
70
|
+
'zfsmgmt:restic_repository',
|
71
|
+
'userrefs',
|
72
|
+
],
|
73
|
+
property_match: { 'zfsmgmt:restic_backup' => 'true' }).each do |blob|
|
74
|
+
zfs,props,zfs_snapshots = blob
|
75
|
+
last_zfs_snapshot = zfs_snapshots.keys.sort { |a,b| zfs_snapshots[a]['creation'] <=> zfs_snapshots[b]['creation'] }.last
|
76
|
+
zfs_snap_time = Time.at(zfs_snapshots[last_zfs_snapshot]['creation'])
|
30
77
|
|
78
|
+
level = 0
|
79
|
+
chain = []
|
80
|
+
zfs_snap_parent = ''
|
81
|
+
restic_snap_parent = ''
|
82
|
+
(restic_snapshots,restic_snapshot_zfs_snapshot_index) = restic_snapshots(zfs,options,props)
|
83
|
+
if restic_snapshot_zfs_snapshot_index.has_key?(last_zfs_snapshot)
|
84
|
+
$logger.warn("backup of this snapshot #{last_zfs_snapshot} already exists in restic, cannot continue with backup of #{zfs}")
|
85
|
+
next # next zfs filesystem to be backed up
|
86
|
+
end
|
87
|
+
if backup_level > 0 and restic_snapshots.count > 0
|
88
|
+
# reverse (oldest first) sorted restic snapshots
|
89
|
+
restic_snap_parent = restic_snapshots.filter { |rsnap|
|
90
|
+
rsnap.has_key?('zfsmgmt:zfs') and rsnap['zfsmgmt:zfs'] == zfs and
|
91
|
+
rsnap.has_key?('zfsmgmt:level') and rsnap['zfsmgmt:level'] < backup_level }.sort {
|
92
|
+
|a,b| a['date_time'] <=> b['date_time'] }.last
|
93
|
+
if restic_snap_parent and
|
94
|
+
zfs_snapshots.has_key?(restic_snap_parent['zfsmgmt:snapshot']) and
|
95
|
+
chain = valid_chain(restic_snap_parent,restic_snapshots,restic_snapshot_zfs_snapshot_index,[]) and
|
96
|
+
chain.length > 0
|
97
|
+
|
98
|
+
level = restic_snap_parent['zfsmgmt:level'] + 1
|
99
|
+
zfs_snap_parent = restic_snap_parent['zfsmgmt:snapshot']
|
100
|
+
$logger.debug("restic_snap_parent: level: #{restic_snap_parent['zfsmgmt:level']} snapshot: #{zfs_snap_parent}")
|
101
|
+
else
|
102
|
+
$logger.error("restic_snap_parent rejected: level: #{restic_snap_parent['zfsmgmt:level']} snapshot: #{restic_snap_parent['zfsmgmt:snapshot']}")
|
103
|
+
end
|
104
|
+
$logger.debug("chain of snapshots: #{chain}")
|
105
|
+
end
|
106
|
+
tags = [ 'zfsmgmt',
|
107
|
+
"zfsmgmt:snapshot=#{last_zfs_snapshot}",
|
108
|
+
"zfsmgmt:zfs=#{zfs}",
|
109
|
+
"zfsmgmt:level=#{level}" ]
|
110
|
+
com = [ ZfsMgmt.global_options['zfs_binary'], 'send', '-L', '-w', '-h', '-p' ]
|
111
|
+
if level > 0
|
112
|
+
if options[:intermediary]
|
113
|
+
com.push('-I')
|
114
|
+
else
|
115
|
+
com.push('-i')
|
116
|
+
end
|
117
|
+
com.push(zfs_snap_parent)
|
118
|
+
tags.push("zfsmgmt:parent=#{zfs_snap_parent}")
|
119
|
+
end
|
120
|
+
com.push( last_zfs_snapshot )
|
121
|
+
com.push( '|', 'mbuffer', '-m', options[:buffer], '-q' )
|
122
|
+
com.push( '|', options[:restic_binary], 'backup', '--stdin',
|
123
|
+
'--stdin-filename', zfs, '--time', "\"#{zfs_snap_time.strftime('%F %T')}\"" )
|
124
|
+
tags.each do |tag|
|
125
|
+
com.push( '--tag', "\"#{tag}\"" )
|
126
|
+
end
|
127
|
+
if options.has_key?('limit_upload')
|
128
|
+
com.push('--limit-upload', options['limit_upload'])
|
129
|
+
end
|
130
|
+
if options.has_key?('password_file')
|
131
|
+
com.push('-p',options['password_file'])
|
132
|
+
end
|
133
|
+
if options.has_key?('repo')
|
134
|
+
com.push('--repo', options['repo'])
|
135
|
+
elsif props.has_key?('zfsmgmt:restic_repository')
|
136
|
+
com.push( '--repo', props['zfsmgmt:restic_repository'] )
|
137
|
+
end
|
138
|
+
if options[:verbose]
|
139
|
+
com.push('--verbose',options[:verbose])
|
140
|
+
elsif $stdout.isatty
|
141
|
+
com.push('-v')
|
142
|
+
end
|
143
|
+
unless ZfsMgmt.zfs_holds(last_zfs_snapshot).include?('zfsmgmt_restic')
|
144
|
+
ZfsMgmt.zfs_hold('zfsmgmt_restic',last_zfs_snapshot)
|
145
|
+
end
|
146
|
+
$logger.info("#{com.join(' ')}")
|
147
|
+
system(com.join(' '))
|
148
|
+
chain_snaps = chain.map do |rsnap|
|
149
|
+
rsnap['zfsmgmt:snapshot']
|
150
|
+
end
|
151
|
+
zfs_snapshots.each do |s,d|
|
152
|
+
d['userrefs'] == 0 and next
|
153
|
+
chain_snaps.include?(s) and next
|
154
|
+
s == last_zfs_snapshot and next
|
155
|
+
if ZfsMgmt.zfs_holds(s).include?('zfsmgmt_restic')
|
156
|
+
ZfsMgmt.zfs_release('zfsmgmt_restic',s)
|
157
|
+
end
|
158
|
+
end
|
159
|
+
end
|
160
|
+
end
|
31
161
|
end
|
32
162
|
|
33
|
-
|
data/lib/zfs_mgmt/version.rb
CHANGED
@@ -0,0 +1,26 @@
|
|
1
|
+
# class list
|
2
|
+
|
3
|
+
class ZfsMgmt::ZfsMgr::List < Thor
|
4
|
+
class_option :filter, :type => :string, :default => '.+',
|
5
|
+
:desc => 'only act on zfs matching this regexp'
|
6
|
+
desc "stale", "list all zfs with stale snapshots"
|
7
|
+
method_option :age, :desc => "timeframe outside of which the zfs will be considered stale", :default => '1d'
|
8
|
+
def stale()
|
9
|
+
ZfsMgmt.global_options = options
|
10
|
+
cutoff = Time.at(Time.now.to_i - ZfsMgmt.timespec_to_seconds(options[:age]))
|
11
|
+
table = Text::Table.new
|
12
|
+
table.head = ['zfs','snapshot','age']
|
13
|
+
table.rows = []
|
14
|
+
ZfsMgmt.zfs_managed_list(filter: options[:filter]).each do |blob|
|
15
|
+
zfs,props,snaps = blob
|
16
|
+
last = snaps.keys.sort { |a,b| snaps[a]['creation'] <=> snaps[b]['creation'] }.last
|
17
|
+
snap_time = Time.at(snaps[last]['creation'])
|
18
|
+
if snap_time < cutoff
|
19
|
+
table.rows << [zfs,last.split('@')[1],snap_time]
|
20
|
+
end
|
21
|
+
end
|
22
|
+
if table.rows.count > 0
|
23
|
+
print table.to_s
|
24
|
+
end
|
25
|
+
end
|
26
|
+
end
|
@@ -0,0 +1,41 @@
|
|
1
|
+
|
2
|
+
class ZfsMgmt::ZfsMgr::Backup < Thor
|
3
|
+
include ZfsMgmt::Restic
|
4
|
+
class_option :filter, :type => :string, :default => '.+',
|
5
|
+
:desc => 'only act on zfs matching this regexp'
|
6
|
+
class_option :restic_binary, :type => :string, :default => 'restic',
|
7
|
+
:desc => 'restic binary'
|
8
|
+
class_option :verbose, :alias => '-v', :type => :numeric,
|
9
|
+
:desc => 'verbosity level for restic'
|
10
|
+
class_option :buffer, :type => :string, :default => '256m',
|
11
|
+
:desc => 'buffer size for mbuffer'
|
12
|
+
class_option :password_file, :alias => '-p', :type => :string,
|
13
|
+
:desc => 'passed to restic'
|
14
|
+
class_option :limit_upload, :type => :numeric,
|
15
|
+
:desc => 'passed to restic'
|
16
|
+
class_option :repo, :type => :string,
|
17
|
+
:desc => 'passed to restic'
|
18
|
+
desc "incremental", "perform incremental backup"
|
19
|
+
method_option :level, :desc => "backup level in integer form", :default => 2, :type => :numeric
|
20
|
+
method_option :intermediary, :alias => '-I', :desc => "pass -I (intermediary) option to zfs send", :default => false, :type => :boolean
|
21
|
+
def incremental()
|
22
|
+
ZfsMgmt.global_options = options
|
23
|
+
ZfsMgmt::Restic.backup(backup_level: options[:level], options: options)
|
24
|
+
end
|
25
|
+
desc "differential", "perform differential backup"
|
26
|
+
method_option :intermediary, :alias => '-I', :desc => "pass -I (intermediary) option to zfs send", :default => false, :type => :boolean
|
27
|
+
def differential()
|
28
|
+
ZfsMgmt.global_options = options
|
29
|
+
ZfsMgmt::Restic.backup(backup_level: 1, options: options)
|
30
|
+
end
|
31
|
+
desc "full", "perform full backup"
|
32
|
+
def full()
|
33
|
+
ZfsMgmt.global_options = options
|
34
|
+
ZfsMgmt::Restic.backup(backup_level: 0, options: options)
|
35
|
+
end
|
36
|
+
end
|
37
|
+
|
38
|
+
class ZfsMgmt::ZfsMgr::Restic < Thor
|
39
|
+
desc "backup SUBCOMMAND ...ARGS", "backup all configured zfs to restic"
|
40
|
+
subcommand "backup", ZfsMgmt::ZfsMgr::Backup
|
41
|
+
end
|
@@ -0,0 +1,27 @@
|
|
1
|
+
# implement snapshot management
|
2
|
+
|
3
|
+
class ZfsMgmt::ZfsMgr::Snapshot < Thor
|
4
|
+
class_option :noop, :type => :boolean, :default => false,
|
5
|
+
:desc => 'pass -n option to zfs commands'
|
6
|
+
class_option :verbose, :type => :boolean, :default => false,
|
7
|
+
:desc => 'pass -v option to zfs commands'
|
8
|
+
class_option :debug, :type => :boolean, :default => false,
|
9
|
+
:desc => 'set logging level to debug'
|
10
|
+
class_option :filter, :type => :string, :default => '.+',
|
11
|
+
:desc => 'only act on zfs matching this regexp'
|
12
|
+
desc "destroy", "apply the snapshot destroy policy to zfs"
|
13
|
+
def destroy()
|
14
|
+
ZfsMgmt.global_options = options
|
15
|
+
ZfsMgmt.snapshot_destroy(noop: options[:noop], verbopt: options[:verbose], debugopt: options[:debug], filter: options[:filter])
|
16
|
+
end
|
17
|
+
desc "policy", "print the policy table for zfs"
|
18
|
+
def policy()
|
19
|
+
ZfsMgmt.global_options = options
|
20
|
+
ZfsMgmt.snapshot_policy(verbopt: options[:verbose], debugopt: options[:debug], filter: options[:filter])
|
21
|
+
end
|
22
|
+
desc "create", "execute zfs snapshot based on zfs properties"
|
23
|
+
def create()
|
24
|
+
ZfsMgmt.global_options = options
|
25
|
+
ZfsMgmt.snapshot_create(verbopt: options[:verbose], debugopt: options[:debug], filter: options[:filter])
|
26
|
+
end
|
27
|
+
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: zfs_mgmt
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.3.
|
4
|
+
version: 0.3.10
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Aran Cox
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2021-02-
|
11
|
+
date: 2021-02-21 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: bundler
|
@@ -127,6 +127,10 @@ files:
|
|
127
127
|
- lib/zfs_mgmt.rb
|
128
128
|
- lib/zfs_mgmt/restic.rb
|
129
129
|
- lib/zfs_mgmt/version.rb
|
130
|
+
- lib/zfs_mgmt/zfs_mgr.rb
|
131
|
+
- lib/zfs_mgmt/zfs_mgr/list.rb
|
132
|
+
- lib/zfs_mgmt/zfs_mgr/restic.rb
|
133
|
+
- lib/zfs_mgmt/zfs_mgr/snapshot.rb
|
130
134
|
- zfs_mgmt.gemspec
|
131
135
|
homepage: https://github.com/aranc23/zfs_mgmt
|
132
136
|
licenses:
|