zfs_mgmt 0.3.3 → 0.3.8
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.gitignore +1 -0
- data/Gemfile +1 -0
- data/Gemfile.lock +3 -1
- data/README.md +59 -23
- data/bin/zfsmgr +8 -24
- data/lib/zfs_mgmt.rb +67 -10
- data/lib/zfs_mgmt/restic.rb +155 -0
- data/lib/zfs_mgmt/version.rb +1 -1
- data/lib/zfs_mgmt/zfs_mgr.rb +6 -0
- data/lib/zfs_mgmt/zfs_mgr/list.rb +25 -0
- data/lib/zfs_mgmt/zfs_mgr/restic.rb +36 -0
- data/lib/zfs_mgmt/zfs_mgr/snapshot.rb +24 -0
- metadata +7 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 2034670d58c6f8eba6eb092109d3f4f5617b70096c2a69430b38bc99f28d2f85
|
4
|
+
data.tar.gz: 2f5e606372b1a915b7294a238bbb8cd230189b5a6e8259e77adcec8d11010112
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: f01390846210b1e9b1bb7e3f800dd970170612641453189adc81edb4f85827ec1abbc22249243661c534bda81197081e57852ac5b812f377f167065095ed3e46
|
7
|
+
data.tar.gz: 17bca1cf6f88bfa37945ee60c9780694ea8f4f2f12bb2a3a4ea6e4a267d3e0abd5ec5d45b5fd777a2dafebf1e73d416578bac6e3fb146431ce045205b12835b7
|
data/.gitignore
CHANGED
data/Gemfile
CHANGED
data/Gemfile.lock
CHANGED
@@ -1,13 +1,14 @@
|
|
1
1
|
PATH
|
2
2
|
remote: .
|
3
3
|
specs:
|
4
|
-
zfs_mgmt (0.3.
|
4
|
+
zfs_mgmt (0.3.8)
|
5
5
|
|
6
6
|
GEM
|
7
7
|
remote: https://rubygems.org/
|
8
8
|
specs:
|
9
9
|
diff-lcs (1.3)
|
10
10
|
filesize (0.2.0)
|
11
|
+
json (2.3.0)
|
11
12
|
rake (13.0.1)
|
12
13
|
rspec (3.9.0)
|
13
14
|
rspec-core (~> 3.9.0)
|
@@ -31,6 +32,7 @@ PLATFORMS
|
|
31
32
|
DEPENDENCIES
|
32
33
|
bundler (~> 1.16)
|
33
34
|
filesize
|
35
|
+
json
|
34
36
|
rake (>= 12.3.3)
|
35
37
|
rspec (~> 3.0)
|
36
38
|
text-table
|
data/README.md
CHANGED
@@ -1,8 +1,11 @@
|
|
1
1
|
# ZfsMgmt
|
2
2
|
|
3
|
-
zfs_mgmt aims to provide some useful helpers for managing zfs
|
3
|
+
zfs_mgmt aims to provide some useful helpers for managing zfs
|
4
|
+
snapshots, and eventually send/recv duties via the zfsmgr script in
|
5
|
+
bin/.
|
4
6
|
|
5
|
-
Currently only snapshot destruction is implemented by a policy
|
7
|
+
Currently only snapshot destruction is implemented by a policy
|
8
|
+
specification stored in zfs properties.
|
6
9
|
|
7
10
|
## Installation
|
8
11
|
|
@@ -16,25 +19,36 @@ Therefore, building the gem and installing, or running ruby inside the src/ dire
|
|
16
19
|
|
17
20
|
## Usage
|
18
21
|
|
19
|
-
The most common usage pattern would be to set zfs properties as
|
22
|
+
The most common usage pattern would be to set zfs properties as
|
23
|
+
explained below, then use **zfsmgr snapshot policy** to print a table
|
24
|
+
of what would be kept and for what reason. Then use **zfsmgr snapshot destroy --noop**
|
25
|
+
to see what would be destroyed, and finally **zfsmgr snapshot destroy**
|
26
|
+
without the --noop option to actually remove snapshots.
|
20
27
|
|
28
|
+
$ zfsmgr
|
21
29
|
Commands:
|
22
30
|
zfsmgr help [COMMAND] # Describe available commands or one specific command
|
31
|
+
zfsmgr list SUBCOMMAND ...ARGS # list filesystems
|
32
|
+
zfsmgr restic SUBCOMMAND ...ARGS # backup zfs to restic
|
23
33
|
zfsmgr snapshot SUBCOMMAND ...ARGS # manage snapshots
|
24
34
|
zfsmgr zfsget [ZFS] # execute zfs get for the given properties and types and parse the output into a nested hash
|
25
|
-
|
26
|
-
|
27
|
-
|
28
|
-
|
29
|
-
|
30
|
-
|
31
|
-
|
32
|
-
|
33
|
-
|
34
|
-
|
35
|
-
|
36
|
-
|
37
|
-
|
35
|
+
|
36
|
+
### list stale
|
37
|
+
|
38
|
+
**zfsmgr list stale will** will list all zfs with "stale" snapshots,
|
39
|
+
that is the newest snapshot is older than 1d or older than the --age
|
40
|
+
parameter.
|
41
|
+
|
42
|
+
### restic backup
|
43
|
+
|
44
|
+
zfsmgr can pipe zfs send output into restic, to allow storing zfs
|
45
|
+
streams in restic repositories. zfsmgr takes an opinionated approach
|
46
|
+
to this task, implementing a traditional full/differential/incremental
|
47
|
+
backup scheme. This should allow a recent snapshot to be restored
|
48
|
+
requiring only 3 restic snapshots to recreate the filesystem. See the
|
49
|
+
properties section for restic related zfs properties.
|
50
|
+
|
51
|
+
$ zfsmgr restic backup full
|
38
52
|
|
39
53
|
|
40
54
|
## Example output
|
@@ -100,20 +114,30 @@ The most common usage pattern would be to set zfs properties as explained below,
|
|
100
114
|
|
101
115
|
## Development
|
102
116
|
|
103
|
-
After checking out the repo, run `bin/setup` to install
|
117
|
+
After checking out the repo, run `bin/setup` to install
|
118
|
+
dependencies. Then, run `rake spec` to run the tests. You can also run
|
119
|
+
`bin/console` for an interactive prompt that will allow you to
|
120
|
+
experiment.
|
104
121
|
|
105
|
-
To install this gem onto your local machine, run `bundle exec rake
|
122
|
+
To install this gem onto your local machine, run `bundle exec rake
|
123
|
+
install`. To release a new version, update the version number in
|
124
|
+
`version.rb`, and then run `bundle exec rake release`, which will
|
125
|
+
create a git tag for the version, push git commits and tags, and push
|
126
|
+
the `.gem` file to [rubygems.org](https://rubygems.org).
|
106
127
|
|
107
128
|
## Contributing
|
108
129
|
|
109
|
-
Bug reports and pull requests are welcome on GitHub at
|
130
|
+
Bug reports and pull requests are welcome on GitHub at
|
131
|
+
https://github.com/aranc23/zfs_mgmt.
|
110
132
|
|
111
133
|
## zfs user properties
|
112
134
|
|
113
|
-
Destruction of zfs snapshots is based on the following zfs user
|
135
|
+
Destruction of zfs snapshots is based on the following zfs user
|
136
|
+
properties:
|
114
137
|
|
115
138
|
### zfsmgmt:manage
|
116
|
-
manage snapshots for this filesystem if this property is 'true'
|
139
|
+
manage snapshots for this filesystem if this property is 'true'
|
140
|
+
(string literal)
|
117
141
|
|
118
142
|
### zfsmgmt:policy
|
119
143
|
|
@@ -127,7 +151,8 @@ Examples:
|
|
127
151
|
- 1y1m1y1d1h (1 of each time frame worth of snapshots)
|
128
152
|
- 72h (72 hourly snapshots)
|
129
153
|
|
130
|
-
The order in which each timeframe is listed in does not matter, and
|
154
|
+
The order in which each timeframe is listed in does not matter, and
|
155
|
+
the supported specs are as follows:
|
131
156
|
|
132
157
|
- h - hourly
|
133
158
|
- d - daily
|
@@ -180,7 +205,11 @@ oldest and unless the property is set to youngest oldest will be used.
|
|
180
205
|
### zfsmgmt:snapshot
|
181
206
|
If this property is 'true' then create a snapshot in the format of
|
182
207
|
zfsmgmt-%FT%T%z. If this property is 'recursive' then create a
|
183
|
-
recursive snapshot of this zfs
|
208
|
+
recursive snapshot of this zfs, but only on zfs where this property is
|
209
|
+
local. If this property is set to the string 'local' and the property
|
210
|
+
is set locally, it will create a snapshot. The intention is that you
|
211
|
+
would use 'local' when you want a zfs snapshot for the filesystem, but
|
212
|
+
NOT it's descendant filesystems.
|
184
213
|
|
185
214
|
### zfsmgmt:snap_prefix
|
186
215
|
Change the zfsmgmt portion of created snapshots, ie: 'autosnap' would
|
@@ -190,6 +219,13 @@ create snapshots called autosnap-%FT%T%z.
|
|
190
219
|
strftime format string used when creating snapshot names, default
|
191
220
|
being %FT%T%z.
|
192
221
|
|
222
|
+
### zfsmgmt:restic_backup
|
223
|
+
boolean, send this zfs to restic
|
224
|
+
|
225
|
+
### zfsmgmt:restic_repository
|
226
|
+
send the zfs to this repository, optional, rely on restic environment
|
227
|
+
variables otherwise
|
228
|
+
|
193
229
|
## Snapshot Management / zfs destroy
|
194
230
|
When destroying snapshots according to a given policy, all snapshots
|
195
231
|
should be considered for deletion and all snapshots should be
|
data/bin/zfsmgr
CHANGED
@@ -1,28 +1,8 @@
|
|
1
1
|
require "thor"
|
2
2
|
require "zfs_mgmt"
|
3
|
-
|
4
|
-
|
5
|
-
|
6
|
-
:desc => 'pass -n option to zfs commands'
|
7
|
-
class_option :verbose, :type => :boolean, :default => false,
|
8
|
-
:desc => 'pass -v option to zfs commands'
|
9
|
-
class_option :debug, :type => :boolean, :default => false,
|
10
|
-
:desc => 'set logging level to debug'
|
11
|
-
class_option :filter, :type => :string, :default => '.+',
|
12
|
-
:desc => 'only act on zfs matching this regexp'
|
13
|
-
desc "destroy", "apply the snapshot destroy policy to zfs"
|
14
|
-
def destroy()
|
15
|
-
ZfsMgmt.snapshot_destroy(noop: options[:noop], verbopt: options[:verbose], debugopt: options[:debug], filter: options[:filter])
|
16
|
-
end
|
17
|
-
desc "policy", "print the policy table for zfs"
|
18
|
-
def policy()
|
19
|
-
ZfsMgmt.snapshot_policy(verbopt: options[:verbose], debugopt: options[:debug], filter: options[:filter])
|
20
|
-
end
|
21
|
-
desc "create", "execute zfs snapshot based on zfs properties"
|
22
|
-
def create()
|
23
|
-
ZfsMgmt.snapshot_create(verbopt: options[:verbose], debugopt: options[:debug], filter: options[:filter])
|
24
|
-
end
|
25
|
-
end
|
3
|
+
require "zfs_mgmt/restic"
|
4
|
+
require "zfs_mgmt/zfs_mgr"
|
5
|
+
require "zfs_mgmt/zfs_mgr/restic"
|
26
6
|
|
27
7
|
class ZfsMgr < Thor
|
28
8
|
desc "zfsget [ZFS]", "execute zfs get for the given properties and types and parse the output into a nested hash"
|
@@ -34,7 +14,11 @@ class ZfsMgr < Thor
|
|
34
14
|
zfs: zfs)
|
35
15
|
end
|
36
16
|
desc "snapshot SUBCOMMAND ...ARGS", "manage snapshots"
|
37
|
-
subcommand "snapshot", Snapshot
|
17
|
+
subcommand "snapshot", ZfsMgmt::ZfsMgr::Snapshot
|
18
|
+
desc "list SUBCOMMAND ...ARGS", "list filesystems"
|
19
|
+
subcommand "list", ZfsMgmt::ZfsMgr::List
|
20
|
+
desc "restic SUBCOMMAND ...ARGS", "backup zfs to restic"
|
21
|
+
subcommand "restic", ZfsMgmt::ZfsMgr::Restic
|
38
22
|
end
|
39
23
|
|
40
24
|
ZfsMgr.start(ARGV)
|
data/lib/zfs_mgmt.rb
CHANGED
@@ -1,5 +1,7 @@
|
|
1
1
|
# coding: utf-8
|
2
2
|
require "zfs_mgmt/version"
|
3
|
+
require "zfs_mgmt/restic"
|
4
|
+
require "zfs_mgmt/zfs_mgr"
|
3
5
|
require 'pp'
|
4
6
|
require 'date'
|
5
7
|
require 'logger'
|
@@ -63,7 +65,45 @@ module ZfsMgmt
|
|
63
65
|
return md[1].to_i
|
64
66
|
end
|
65
67
|
end
|
66
|
-
|
68
|
+
|
69
|
+
def self.zfs_holds(snapshot)
|
70
|
+
com = ['zfs', 'holds', '-H', snapshot]
|
71
|
+
$logger.debug("#{com.join(' ')}")
|
72
|
+
out = %x(#{com.join(' ')})
|
73
|
+
unless $?.success?
|
74
|
+
errstr = "unable to retrieves holds for snapshot: #{snapshot}"
|
75
|
+
$logger.error(errstr)
|
76
|
+
raise errstr
|
77
|
+
end
|
78
|
+
a = []
|
79
|
+
out.split("\n").each do |ln|
|
80
|
+
a.push(ln.split("\t")[1])
|
81
|
+
end
|
82
|
+
a
|
83
|
+
end
|
84
|
+
|
85
|
+
def self.zfs_hold(hold,snapshot)
|
86
|
+
com = ['zfs', 'hold', hold, snapshot]
|
87
|
+
$logger.debug("#{com.join(' ')}")
|
88
|
+
system(com.join(' '))
|
89
|
+
unless $?.success?
|
90
|
+
errstr = "unable to set hold: #{hold} for snapshot: #{snapshot}"
|
91
|
+
$logger.error(errstr)
|
92
|
+
raise errstr
|
93
|
+
end
|
94
|
+
end
|
95
|
+
|
96
|
+
def self.zfs_release(hold,snapshot)
|
97
|
+
com = ['zfs', 'release', hold, snapshot]
|
98
|
+
$logger.debug("#{com.join(' ')}")
|
99
|
+
system(com.join(' '))
|
100
|
+
unless $?.success?
|
101
|
+
errstr = "unable to release hold: #{hold} for snapshot: #{snapshot}"
|
102
|
+
$logger.error(errstr)
|
103
|
+
raise errstr
|
104
|
+
end
|
105
|
+
end
|
106
|
+
|
67
107
|
def self.zfsget(properties: ['name'],types: ['filesystem','volume'],zfs: '')
|
68
108
|
results={}
|
69
109
|
com = ['zfs', 'get', '-Hp', properties.join(','), '-t', types.join(','), zfs]
|
@@ -191,24 +231,25 @@ module ZfsMgmt
|
|
191
231
|
}
|
192
232
|
return saved,saved_snaps,deleteme
|
193
233
|
end
|
194
|
-
def self.zfs_managed_list(filter: '.+')
|
234
|
+
def self.zfs_managed_list(filter: '.+', properties: custom_properties(), property_match: { 'zfsmgmt:manage' => 'true' } )
|
195
235
|
zfss = [] # array of arrays
|
196
|
-
zfsget(properties:
|
236
|
+
zfsget(properties: properties).each do |zfs,props|
|
197
237
|
unless /#{filter}/ =~ zfs
|
198
238
|
next
|
199
239
|
end
|
200
|
-
|
201
|
-
|
240
|
+
managed = true
|
241
|
+
property_match.each do |k,v|
|
242
|
+
unless props.has_key?(k) and props[k] == v
|
243
|
+
managed = false
|
244
|
+
break
|
245
|
+
end
|
202
246
|
end
|
247
|
+
next unless managed
|
203
248
|
snaps = self.zfsget(properties: ['name','creation','userrefs','used','written','referenced'],types: ['snapshot'], zfs: zfs)
|
204
249
|
if snaps.length == 0
|
205
250
|
$logger.warn("unable to process this zfs, no snapshots at all: #{zfs}")
|
206
251
|
next
|
207
252
|
end
|
208
|
-
unless props.has_key?('zfsmgmt:policy') and policy = policy_parser(props['zfsmgmt:policy'])
|
209
|
-
$logger.error("zfs_mgmt is configured to manage #{zfs}, but there is no valid policy configuration, skipping")
|
210
|
-
next # zfs
|
211
|
-
end
|
212
253
|
zfss.push([zfs,props,snaps])
|
213
254
|
end
|
214
255
|
return zfss
|
@@ -221,6 +262,10 @@ module ZfsMgmt
|
|
221
262
|
end
|
222
263
|
zfs_managed_list(filter: filter).each do |zdata|
|
223
264
|
(zfs,props,snaps) = zdata
|
265
|
+
unless props.has_key?('zfsmgmt:policy') and policy_parser(props['zfsmgmt:policy'])
|
266
|
+
$logger.error("zfs_mgmt is configured to manage #{zfs}, but there is no valid policy configuration, skipping")
|
267
|
+
next # zfs
|
268
|
+
end
|
224
269
|
# call the function that decides who to save and who to delete
|
225
270
|
(saved,saved_snaps,deleteme) = snapshot_destroy_policy(zfs,props,snaps)
|
226
271
|
|
@@ -246,6 +291,11 @@ module ZfsMgmt
|
|
246
291
|
end
|
247
292
|
zfs_managed_list(filter: filter).each do |zdata|
|
248
293
|
(zfs,props,snaps) = zdata
|
294
|
+
unless props.has_key?('zfsmgmt:policy') and policy_parser(props['zfsmgmt:policy'])
|
295
|
+
$logger.error("zfs_mgmt is configured to manage #{zfs}, but there is no valid policy configuration, skipping")
|
296
|
+
next # zfs
|
297
|
+
end
|
298
|
+
|
249
299
|
# call the function that decides who to save and who to delete
|
250
300
|
(saved,saved_snaps,deleteme) = snapshot_destroy_policy(zfs,props,snaps)
|
251
301
|
|
@@ -310,8 +360,15 @@ module ZfsMgmt
|
|
310
360
|
end
|
311
361
|
dt = DateTime.now
|
312
362
|
zfsget(properties: custom_properties()).each do |zfs,props|
|
363
|
+
unless /#{filter}/ =~ zfs
|
364
|
+
next
|
365
|
+
end
|
313
366
|
# zfs must have snapshot set to true or recursive
|
314
|
-
if props.has_key?('zfsmgmt:snapshot') and
|
367
|
+
if props.has_key?('zfsmgmt:snapshot') and
|
368
|
+
props['zfsmgmt:snapshot'] == 'true' or
|
369
|
+
( props['zfsmgmt:snapshot'] == 'recursive' and props['zfsmgmt:snapshot@source'] == 'local' ) or
|
370
|
+
( props['zfsmgmt:snapshot'] == 'local' and props['zfsmgmt:snapshot@source'] == 'local' )
|
371
|
+
|
315
372
|
prefix = ( props.has_key?('zfsmgmt:snap_prefix') ? props['zfsmgmt:snap_prefix'] : 'zfsmgmt' )
|
316
373
|
ts = ( props.has_key?('zfsmgmt:snap_timestamp') ? props['zfsmgmt:snap_timestamp'] : '%FT%T%z' )
|
317
374
|
com = ['zfs','snapshot']
|
@@ -0,0 +1,155 @@
|
|
1
|
+
require "json"
|
2
|
+
|
3
|
+
module ZfsMgmt::Restic
|
4
|
+
def self.restic_snapshots(zfs,options,props)
|
5
|
+
# query the restic database
|
6
|
+
com = [ options[:restic_binary],
|
7
|
+
'snapshots',
|
8
|
+
'--json',
|
9
|
+
'--tag', 'zfsmgmt',
|
10
|
+
'--path', "/#{zfs}",
|
11
|
+
]
|
12
|
+
if options.has_key?('password_file')
|
13
|
+
com.push('-p',options['password_file'])
|
14
|
+
end
|
15
|
+
if props.has_key?('zfsmgmt:restic_repository')
|
16
|
+
com.push( '-r', props['zfsmgmt:restic_repository'] )
|
17
|
+
end
|
18
|
+
|
19
|
+
$logger.debug("#{com.join(' ')}")
|
20
|
+
restic_output = %x(#{com.join(' ')})
|
21
|
+
unless $?.success?
|
22
|
+
$logger.error("unable to query the restic database")
|
23
|
+
raise "unable to query the restic database"
|
24
|
+
end
|
25
|
+
restic_snapshots = JSON.parse(restic_output)
|
26
|
+
restic_snapshot_zfs_snapshot_index = {}
|
27
|
+
restic_snapshots.each do |snappy|
|
28
|
+
snappy['date_time'] = DateTime.parse(snappy['time'])
|
29
|
+
if snappy.has_key?('tags')
|
30
|
+
snappy['tags'].each do |t|
|
31
|
+
if m = /^(zfsmgmt:.+?)=(.+)/.match(t)
|
32
|
+
if ['zfsmgmt:level'].include?(m[1])
|
33
|
+
snappy[m[1]] = m[2].to_i
|
34
|
+
else
|
35
|
+
snappy[m[1]] = m[2]
|
36
|
+
end
|
37
|
+
if m[1] == 'zfsmgmt:snapshot'
|
38
|
+
restic_snapshot_zfs_snapshot_index[m[2]] = snappy
|
39
|
+
end
|
40
|
+
end
|
41
|
+
end
|
42
|
+
end
|
43
|
+
end
|
44
|
+
return([restic_snapshots,restic_snapshot_zfs_snapshot_index])
|
45
|
+
end
|
46
|
+
|
47
|
+
def self.valid_chain(snap,restic_snapshots,restic_snapshot_zfs_snapshot_index,a)
|
48
|
+
if snap['zfsmgmt:level'] == 0
|
49
|
+
a.push(snap)
|
50
|
+
$logger.debug("found complete chain culminating in full backup of: #{snap['zfsmgmt:snapshot']}")
|
51
|
+
return a
|
52
|
+
elsif restic_snapshot_zfs_snapshot_index.has_key?(snap['zfsmgmt:parent'])
|
53
|
+
a.push(snap)
|
54
|
+
$logger.debug("found another link in the chain: #{snap['zfsmgmt:snapshot']} => #{snap['zfsmgmt:parent']}")
|
55
|
+
return valid_chain(restic_snapshot_zfs_snapshot_index[snap['zfsmgmt:parent']],restic_snapshots,restic_snapshot_zfs_snapshot_index,a)
|
56
|
+
else
|
57
|
+
$logger.error("broken chain: looking for the parent of #{snap['zfsmgmt:snapshot']} (#{snap['zfsmgmt:parent']}) and failed to find")
|
58
|
+
return []
|
59
|
+
end
|
60
|
+
end
|
61
|
+
|
62
|
+
|
63
|
+
def self.backup(backup_level: 2,
|
64
|
+
options: {})
|
65
|
+
ZfsMgmt.zfs_managed_list(filter: options['filter'],
|
66
|
+
properties: ['name',
|
67
|
+
'zfsmgmt:restic_backup',
|
68
|
+
'zfsmgmt:restic_repository',
|
69
|
+
'userrefs',
|
70
|
+
],
|
71
|
+
property_match: { 'zfsmgmt:restic_backup' => 'true' }).each do |blob|
|
72
|
+
zfs,props,zfs_snapshots = blob
|
73
|
+
last_zfs_snapshot = zfs_snapshots.keys.sort { |a,b| zfs_snapshots[a]['creation'] <=> zfs_snapshots[b]['creation'] }.last
|
74
|
+
zfs_snap_time = Time.at(zfs_snapshots[last_zfs_snapshot]['creation'])
|
75
|
+
|
76
|
+
level = 0
|
77
|
+
chain = []
|
78
|
+
zfs_snap_parent = ''
|
79
|
+
restic_snap_parent = ''
|
80
|
+
(restic_snapshots,restic_snapshot_zfs_snapshot_index) = restic_snapshots(zfs,options,props)
|
81
|
+
if restic_snapshot_zfs_snapshot_index.has_key?(last_zfs_snapshot)
|
82
|
+
$logger.warn("backup of this snapshot #{last_zfs_snapshot} already exists in restic, cannot continue with backup of #{zfs}")
|
83
|
+
next # next zfs filesystem to be backed up
|
84
|
+
end
|
85
|
+
if backup_level > 0 and restic_snapshots.count > 0
|
86
|
+
# reverse (oldest first) sorted restic snapshots
|
87
|
+
restic_snap_parent = restic_snapshots.filter { |rsnap|
|
88
|
+
rsnap.has_key?('zfsmgmt:zfs') and rsnap['zfsmgmt:zfs'] == zfs and
|
89
|
+
rsnap.has_key?('zfsmgmt:level') and rsnap['zfsmgmt:level'] < backup_level }.sort {
|
90
|
+
|a,b| a['date_time'] <=> b['date_time'] }.last
|
91
|
+
if restic_snap_parent and
|
92
|
+
zfs_snapshots.has_key?(restic_snap_parent['zfsmgmt:snapshot']) and
|
93
|
+
chain = valid_chain(restic_snap_parent,restic_snapshots,restic_snapshot_zfs_snapshot_index,[]) and
|
94
|
+
chain.length > 0
|
95
|
+
|
96
|
+
level = restic_snap_parent['zfsmgmt:level'] + 1
|
97
|
+
zfs_snap_parent = restic_snap_parent['zfsmgmt:snapshot']
|
98
|
+
$logger.debug("restic_snap_parent: level: #{restic_snap_parent['zfsmgmt:level']} snapshot: #{zfs_snap_parent}")
|
99
|
+
else
|
100
|
+
$logger.error("restic_snap_parent rejected: level: #{restic_snap_parent['zfsmgmt:level']} snapshot: #{restic_snap_parent['zfsmgmt:snapshot']}")
|
101
|
+
end
|
102
|
+
$logger.debug("chain of snapshots: #{chain}")
|
103
|
+
end
|
104
|
+
tags = [ 'zfsmgmt',
|
105
|
+
"zfsmgmt:snapshot=#{last_zfs_snapshot}",
|
106
|
+
"zfsmgmt:zfs=#{zfs}",
|
107
|
+
"zfsmgmt:level=#{level}" ]
|
108
|
+
com = [ options[:zfs_binary], 'send', '-L', '-w', '-h', '-p' ]
|
109
|
+
if level > 0
|
110
|
+
if options[:intermediary]
|
111
|
+
com.push('-I')
|
112
|
+
else
|
113
|
+
com.push('-i')
|
114
|
+
end
|
115
|
+
com.push(zfs_snap_parent)
|
116
|
+
tags.push("zfsmgmt:parent=#{zfs_snap_parent}")
|
117
|
+
end
|
118
|
+
com.push( last_zfs_snapshot )
|
119
|
+
com.push( '|', 'mbuffer', '-m', options[:buffer], '-q' )
|
120
|
+
com.push( '|', options[:restic_binary], 'backup', '--stdin',
|
121
|
+
'--stdin-filename', zfs, '--time', "\"#{zfs_snap_time.strftime('%F %T')}\"" )
|
122
|
+
tags.each do |tag|
|
123
|
+
com.push( '--tag', "\"#{tag}\"" )
|
124
|
+
end
|
125
|
+
if options.has_key?('password_file')
|
126
|
+
com.push('-p',options['password_file'])
|
127
|
+
end
|
128
|
+
if props.has_key?('zfsmgmt:restic_repository')
|
129
|
+
com.push( '-r', props['zfsmgmt:restic_repository'] )
|
130
|
+
end
|
131
|
+
if options[:verbose]
|
132
|
+
com.push('--verbose',options[:verbose])
|
133
|
+
elsif $stdout.isatty
|
134
|
+
com.push('-v')
|
135
|
+
end
|
136
|
+
unless ZfsMgmt.zfs_holds(last_zfs_snapshot).include?('zfsmgmt_restic')
|
137
|
+
ZfsMgmt.zfs_hold('zfsmgmt_restic',last_zfs_snapshot)
|
138
|
+
end
|
139
|
+
$logger.info("#{com.join(' ')}")
|
140
|
+
system(com.join(' '))
|
141
|
+
chain_snaps = chain.map do |rsnap|
|
142
|
+
rsnap['zfsmgmt:snapshot']
|
143
|
+
end
|
144
|
+
zfs_snapshots.each do |s,d|
|
145
|
+
d['userrefs'] == 0 and next
|
146
|
+
chain_snaps.include?(s) and next
|
147
|
+
s == last_zfs_snapshot and next
|
148
|
+
if ZfsMgmt.zfs_holds(s).include?('zfsmgmt_restic')
|
149
|
+
ZfsMgmt.zfs_release('zfsmgmt_restic',s)
|
150
|
+
end
|
151
|
+
end
|
152
|
+
end
|
153
|
+
end
|
154
|
+
end
|
155
|
+
|
data/lib/zfs_mgmt/version.rb
CHANGED
@@ -0,0 +1,25 @@
|
|
1
|
+
# class list
|
2
|
+
|
3
|
+
class ZfsMgmt::ZfsMgr::List < Thor
|
4
|
+
class_option :filter, :type => :string, :default => '.+',
|
5
|
+
:desc => 'only act on zfs matching this regexp'
|
6
|
+
desc "stale", "list all zfs with stale snapshots"
|
7
|
+
method_option :age, :desc => "timeframe outside of which the zfs will be considered stale", :default => '1d'
|
8
|
+
def stale()
|
9
|
+
cutoff = Time.at(Time.now.to_i - ZfsMgmt.timespec_to_seconds(options[:age]))
|
10
|
+
table = Text::Table.new
|
11
|
+
table.head = ['zfs','snapshot','age']
|
12
|
+
table.rows = []
|
13
|
+
ZfsMgmt.zfs_managed_list(filter: options[:filter]).each do |blob|
|
14
|
+
zfs,props,snaps = blob
|
15
|
+
last = snaps.keys.sort { |a,b| snaps[a]['creation'] <=> snaps[b]['creation'] }.last
|
16
|
+
snap_time = Time.at(snaps[last]['creation'])
|
17
|
+
if snap_time < cutoff
|
18
|
+
table.rows << [zfs,last.split('@')[1],snap_time]
|
19
|
+
end
|
20
|
+
end
|
21
|
+
if table.rows.count > 0
|
22
|
+
print table.to_s
|
23
|
+
end
|
24
|
+
end
|
25
|
+
end
|
@@ -0,0 +1,36 @@
|
|
1
|
+
|
2
|
+
class ZfsMgmt::ZfsMgr::Backup < Thor
|
3
|
+
include ZfsMgmt::Restic
|
4
|
+
class_option :filter, :type => :string, :default => '.+',
|
5
|
+
:desc => 'only act on zfs matching this regexp'
|
6
|
+
class_option :restic_binary, :type => :string, :default => 'restic',
|
7
|
+
:desc => 'restic binary'
|
8
|
+
class_option :zfs_binary, :type => :string, :default => 'zfs',
|
9
|
+
:desc => 'zfs binary'
|
10
|
+
class_option :verbose, :alias => '-v', :type => :numeric,
|
11
|
+
:desc => 'verbosity level for restic'
|
12
|
+
class_option :buffer, :type => :string, :default => '256m',
|
13
|
+
:desc => 'buffer size for mbuffer'
|
14
|
+
class_option :password_file, :alias => '-p', :type => :string,
|
15
|
+
:desc => 'passed to restic'
|
16
|
+
desc "incremental", "perform incremental backup"
|
17
|
+
method_option :level, :desc => "backup level in integer form", :default => 2, :type => :numeric
|
18
|
+
method_option :intermediary, :alias => '-I', :desc => "pass -I (intermediary) option to zfs send", :default => false, :type => :boolean
|
19
|
+
def incremental()
|
20
|
+
ZfsMgmt::Restic.backup(backup_level: options[:level], options: options)
|
21
|
+
end
|
22
|
+
desc "differential", "perform differential backup"
|
23
|
+
method_option :intermediary, :alias => '-I', :desc => "pass -I (intermediary) option to zfs send", :default => false, :type => :boolean
|
24
|
+
def differential()
|
25
|
+
ZfsMgmt::Restic.backup(backup_level: 1, options: options)
|
26
|
+
end
|
27
|
+
desc "full", "perform full backup"
|
28
|
+
def full()
|
29
|
+
ZfsMgmt::Restic.backup(backup_level: 0, options: options)
|
30
|
+
end
|
31
|
+
end
|
32
|
+
|
33
|
+
class ZfsMgmt::ZfsMgr::Restic < Thor
|
34
|
+
desc "backup SUBCOMMAND ...ARGS", "backup all configured zfs to restic"
|
35
|
+
subcommand "backup", ZfsMgmt::ZfsMgr::Backup
|
36
|
+
end
|
@@ -0,0 +1,24 @@
|
|
1
|
+
# implement snapshot management
|
2
|
+
|
3
|
+
class ZfsMgmt::ZfsMgr::Snapshot < Thor
|
4
|
+
class_option :noop, :type => :boolean, :default => false,
|
5
|
+
:desc => 'pass -n option to zfs commands'
|
6
|
+
class_option :verbose, :type => :boolean, :default => false,
|
7
|
+
:desc => 'pass -v option to zfs commands'
|
8
|
+
class_option :debug, :type => :boolean, :default => false,
|
9
|
+
:desc => 'set logging level to debug'
|
10
|
+
class_option :filter, :type => :string, :default => '.+',
|
11
|
+
:desc => 'only act on zfs matching this regexp'
|
12
|
+
desc "destroy", "apply the snapshot destroy policy to zfs"
|
13
|
+
def destroy()
|
14
|
+
ZfsMgmt.snapshot_destroy(noop: options[:noop], verbopt: options[:verbose], debugopt: options[:debug], filter: options[:filter])
|
15
|
+
end
|
16
|
+
desc "policy", "print the policy table for zfs"
|
17
|
+
def policy()
|
18
|
+
ZfsMgmt.snapshot_policy(verbopt: options[:verbose], debugopt: options[:debug], filter: options[:filter])
|
19
|
+
end
|
20
|
+
desc "create", "execute zfs snapshot based on zfs properties"
|
21
|
+
def create()
|
22
|
+
ZfsMgmt.snapshot_create(verbopt: options[:verbose], debugopt: options[:debug], filter: options[:filter])
|
23
|
+
end
|
24
|
+
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: zfs_mgmt
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.3.
|
4
|
+
version: 0.3.8
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Aran Cox
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date:
|
11
|
+
date: 2021-02-18 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: bundler
|
@@ -125,7 +125,12 @@ files:
|
|
125
125
|
- bin/zfssendman
|
126
126
|
- bin/zfssnapman
|
127
127
|
- lib/zfs_mgmt.rb
|
128
|
+
- lib/zfs_mgmt/restic.rb
|
128
129
|
- lib/zfs_mgmt/version.rb
|
130
|
+
- lib/zfs_mgmt/zfs_mgr.rb
|
131
|
+
- lib/zfs_mgmt/zfs_mgr/list.rb
|
132
|
+
- lib/zfs_mgmt/zfs_mgr/restic.rb
|
133
|
+
- lib/zfs_mgmt/zfs_mgr/snapshot.rb
|
129
134
|
- zfs_mgmt.gemspec
|
130
135
|
homepage: https://github.com/aranc23/zfs_mgmt
|
131
136
|
licenses:
|