zfs_mgmt 0.3.5 → 0.3.10

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: bce5f90411d08b54334f8694e7956f718c1e63995f2c799b95cb170b302c2c8a
4
- data.tar.gz: 1d82a41657f40fe371814210be392d8c34be333816530fcff6a67ae778dc120b
3
+ metadata.gz: a54ed2e0ad255f69367aaeb25fd0dddcc8346cec0fd8be02561e4ea1210510f2
4
+ data.tar.gz: 1d31c311f63835c3f7a2b6e771b06628f1901b62130e9ec5d0255ff987fb89bc
5
5
  SHA512:
6
- metadata.gz: e500b4cc3b155528c470c5aa26338a948e783a5f951a7a5e0d6c05b6739c07cccd6bfe0a1dd001a34722f47e611cc85e2f931d8e3cbca58e37b0f3b0d23606da
7
- data.tar.gz: d3ca6969f760260449f85d99e05133f542700f9d0322a84f771cd157d3ec0729c2e86dc33f0326c53b1f35065cb28b879bbad85c6e422c4bae95b20cc9d21f06
6
+ metadata.gz: c0534143034e6edb6ce1161bacd86765cf9a151812d1849f3570c983691dfdf8d03f5d0aeefb070fa144fa0fa6838711b855e07180a7c70cd14f3f00d561d562
7
+ data.tar.gz: dbf53bcfe6223106f080393621d58ef538a2cf9fe9ae9e7f338b2cbe4e5d88d5bca2fcf44f3c6e8ffb76050ce2d0cfc75fa12638b8d19aae759b9b507fab3a65
data/Gemfile CHANGED
@@ -6,4 +6,5 @@ git_source(:github) {|repo_name| "https://github.com/#{repo_name}" }
6
6
  gem "thor"
7
7
  gem "text-table"
8
8
  gem "filesize"
9
+ gem "json"
9
10
  gemspec
data/Gemfile.lock CHANGED
@@ -1,13 +1,14 @@
1
1
  PATH
2
2
  remote: .
3
3
  specs:
4
- zfs_mgmt (0.3.5)
4
+ zfs_mgmt (0.3.10)
5
5
 
6
6
  GEM
7
7
  remote: https://rubygems.org/
8
8
  specs:
9
9
  diff-lcs (1.3)
10
10
  filesize (0.2.0)
11
+ json (2.3.0)
11
12
  rake (13.0.1)
12
13
  rspec (3.9.0)
13
14
  rspec-core (~> 3.9.0)
@@ -31,6 +32,7 @@ PLATFORMS
31
32
  DEPENDENCIES
32
33
  bundler (~> 1.16)
33
34
  filesize
35
+ json
34
36
  rake (>= 12.3.3)
35
37
  rspec (~> 3.0)
36
38
  text-table
data/README.md CHANGED
@@ -1,8 +1,11 @@
1
1
  # ZfsMgmt
2
2
 
3
- zfs_mgmt aims to provide some useful helpers for managing zfs snapshots, and eventually send/recv duties via the zfsmgr script in bin/.
3
+ zfs_mgmt aims to provide some useful helpers for managing zfs
4
+ snapshots, and eventually send/recv duties via the zfsmgr script in
5
+ bin/.
4
6
 
5
- Currently only snapshot destruction is implemented by a policy specification stored in zfs properties.
7
+ Currently only snapshot destruction is implemented by a policy
8
+ specification stored in zfs properties.
6
9
 
7
10
  ## Installation
8
11
 
@@ -16,25 +19,36 @@ Therefore, building the gem and installing, or running ruby inside the src/ dire
16
19
 
17
20
  ## Usage
18
21
 
19
- The most common usage pattern would be to set zfs properties as explained below, then use **zfsmgr snapshot policy** to print a table of what would be kept and for what reason. Then use **zfsmgr snapshot destroy --noop** to see what would be destroyed, and finally **zfsmgr snapshot destroy** without the --noop option to actually remove snapshots.
22
+ The most common usage pattern would be to set zfs properties as
23
+ explained below, then use **zfsmgr snapshot policy** to print a table
24
+ of what would be kept and for what reason. Then use **zfsmgr snapshot destroy --noop**
25
+ to see what would be destroyed, and finally **zfsmgr snapshot destroy**
26
+ without the --noop option to actually remove snapshots.
20
27
 
28
+ $ zfsmgr
21
29
  Commands:
22
30
  zfsmgr help [COMMAND] # Describe available commands or one specific command
31
+ zfsmgr list SUBCOMMAND ...ARGS # list filesystems
32
+ zfsmgr restic SUBCOMMAND ...ARGS # backup zfs to restic
23
33
  zfsmgr snapshot SUBCOMMAND ...ARGS # manage snapshots
24
34
  zfsmgr zfsget [ZFS] # execute zfs get for the given properties and types and parse the output into a nested hash
25
-
26
-
27
- zfsmgr snapshot create # execute zfs snapshot based on zfs properties
28
- zfsmgr snapshot destroy # apply the snapshot destroy policy to zfs
29
- zfsmgr snapshot help [COMMAND] # Describe subcommands or one specific subcommand
30
- zfsmgr snapshot policy # print the policy table for zfs
31
-
32
- Options:
33
- [--noop], [--no-noop] # pass -n option to zfs commands
34
- [--verbose], [--no-verbose] # pass -v option to zfs commands
35
- [--debug], [--no-debug] # set logging level to debug
36
- [--filter=FILTER] # only act on zfs matching this regexp
37
- # Default: .+
35
+
36
+ ### list stale
37
+
38
+ **zfsmgr list stale will** will list all zfs with "stale" snapshots,
39
+ that is the newest snapshot is older than 1d or older than the --age
40
+ parameter.
41
+
42
+ ### restic backup
43
+
44
+ zfsmgr can pipe zfs send output into restic, to allow storing zfs
45
+ streams in restic repositories. zfsmgr takes an opinionated approach
46
+ to this task, implementing a traditional full/differential/incremental
47
+ backup scheme. This should allow a recent snapshot to be restored
48
+ requiring only 3 restic snapshots to recreate the filesystem. See the
49
+ properties section for restic related zfs properties.
50
+
51
+ $ zfsmgr restic backup full
38
52
 
39
53
 
40
54
  ## Example output
@@ -100,20 +114,30 @@ The most common usage pattern would be to set zfs properties as explained below,
100
114
 
101
115
  ## Development
102
116
 
103
- After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
117
+ After checking out the repo, run `bin/setup` to install
118
+ dependencies. Then, run `rake spec` to run the tests. You can also run
119
+ `bin/console` for an interactive prompt that will allow you to
120
+ experiment.
104
121
 
105
- To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
122
+ To install this gem onto your local machine, run `bundle exec rake
123
+ install`. To release a new version, update the version number in
124
+ `version.rb`, and then run `bundle exec rake release`, which will
125
+ create a git tag for the version, push git commits and tags, and push
126
+ the `.gem` file to [rubygems.org](https://rubygems.org).
106
127
 
107
128
  ## Contributing
108
129
 
109
- Bug reports and pull requests are welcome on GitHub at https://github.com/aranc23/zfs_mgmt.
130
+ Bug reports and pull requests are welcome on GitHub at
131
+ https://github.com/aranc23/zfs_mgmt.
110
132
 
111
133
  ## zfs user properties
112
134
 
113
- Destruction of zfs snapshots is based on the following zfs user properties:
135
+ Destruction of zfs snapshots is based on the following zfs user
136
+ properties:
114
137
 
115
138
  ### zfsmgmt:manage
116
- manage snapshots for this filesystem if this property is 'true' (string literal)
139
+ manage snapshots for this filesystem if this property is 'true'
140
+ (string literal)
117
141
 
118
142
  ### zfsmgmt:policy
119
143
 
@@ -127,7 +151,8 @@ Examples:
127
151
  - 1y1m1y1d1h (1 of each time frame worth of snapshots)
128
152
  - 72h (72 hourly snapshots)
129
153
 
130
- The order in which each timeframe is listed in does not matter, and the supported specs are as follows:
154
+ The order in which each timeframe is listed in does not matter, and
155
+ the supported specs are as follows:
131
156
 
132
157
  - h - hourly
133
158
  - d - daily
@@ -194,6 +219,13 @@ create snapshots called autosnap-%FT%T%z.
194
219
  strftime format string used when creating snapshot names, default
195
220
  being %FT%T%z.
196
221
 
222
+ ### zfsmgmt:restic_backup
223
+ boolean, send this zfs to restic
224
+
225
+ ### zfsmgmt:restic_repository
226
+ send the zfs to this repository, optional, rely on restic environment
227
+ variables otherwise
228
+
197
229
  ## Snapshot Management / zfs destroy
198
230
  When destroying snapshots according to a given policy, all snapshots
199
231
  should be considered for deletion and all snapshots should be
data/bin/zfsmgr CHANGED
@@ -1,44 +1,29 @@
1
1
  require "thor"
2
2
  require "zfs_mgmt"
3
-
4
- class Snapshot < Thor
5
- class_option :noop, :type => :boolean, :default => false,
6
- :desc => 'pass -n option to zfs commands'
7
- class_option :verbose, :type => :boolean, :default => false,
8
- :desc => 'pass -v option to zfs commands'
9
- class_option :debug, :type => :boolean, :default => false,
10
- :desc => 'set logging level to debug'
11
- class_option :filter, :type => :string, :default => '.+',
12
- :desc => 'only act on zfs matching this regexp'
13
- desc "destroy", "apply the snapshot destroy policy to zfs"
14
- def destroy()
15
- ZfsMgmt.snapshot_destroy(noop: options[:noop], verbopt: options[:verbose], debugopt: options[:debug], filter: options[:filter])
16
- end
17
- desc "policy", "print the policy table for zfs"
18
- def policy()
19
- ZfsMgmt.snapshot_policy(verbopt: options[:verbose], debugopt: options[:debug], filter: options[:filter])
20
- end
21
- desc "create", "execute zfs snapshot based on zfs properties"
22
- def create()
23
- ZfsMgmt.snapshot_create(verbopt: options[:verbose], debugopt: options[:debug], filter: options[:filter])
24
- end
25
- end
3
+ require "zfs_mgmt/restic"
4
+ require "zfs_mgmt/zfs_mgr"
5
+ require "zfs_mgmt/zfs_mgr/restic"
26
6
 
27
7
  class ZfsMgr < Thor
8
+ class_option :zfs_binary, :type => :string, :default => 'zfs',
9
+ :desc => 'zfs binary'
10
+ class_option :zpool_binary, :type => :string, :default => 'zpool',
11
+ :desc => 'zfs binary'
28
12
  desc "zfsget [ZFS]", "execute zfs get for the given properties and types and parse the output into a nested hash"
29
13
  method_option :properties, :type => :array, :default => ['name'], :desc => "List of properties passed to zfs get"
30
14
  method_option :types, :type => :array, :default => ['filesystem','volume'], enum: ['filesystem','volume','snapshot'], :desc => "list of types"
31
15
  def zfsget(zfs)
16
+ ZfsMgmt.global_options = options
32
17
  pp ZfsMgmt.zfsget(properties: options[:properties],
33
18
  types: options[:types],
34
19
  zfs: zfs)
35
20
  end
36
21
  desc "snapshot SUBCOMMAND ...ARGS", "manage snapshots"
37
- subcommand "snapshot", Snapshot
22
+ subcommand "snapshot", ZfsMgmt::ZfsMgr::Snapshot
38
23
  desc "list SUBCOMMAND ...ARGS", "list filesystems"
39
- subcommand "list", ZfsMgmtList
24
+ subcommand "list", ZfsMgmt::ZfsMgr::List
40
25
  desc "restic SUBCOMMAND ...ARGS", "backup zfs to restic"
41
- subcommand "restic", ZfsMgmtRestic
26
+ subcommand "restic", ZfsMgmt::ZfsMgr::Restic
42
27
  end
43
28
 
44
29
  ZfsMgr.start(ARGV)
data/lib/zfs_mgmt.rb CHANGED
@@ -1,6 +1,7 @@
1
1
  # coding: utf-8
2
2
  require "zfs_mgmt/version"
3
3
  require "zfs_mgmt/restic"
4
+ require "zfs_mgmt/zfs_mgr"
4
5
  require 'pp'
5
6
  require 'date'
6
7
  require 'logger'
@@ -37,6 +38,9 @@ $properties_xlate = {
37
38
  }
38
39
 
39
40
  module ZfsMgmt
41
+ class << self
42
+ attr_accessor :global_options
43
+ end
40
44
  def self.custom_properties()
41
45
  return [
42
46
  'policy',
@@ -64,10 +68,49 @@ module ZfsMgmt
64
68
  return md[1].to_i
65
69
  end
66
70
  end
67
-
71
+
72
+ def self.zfs_holds(snapshot)
73
+ com = [global_options['zfs_binary'], 'holds', '-H', snapshot]
74
+ $logger.debug("#{com.join(' ')}")
75
+ out = %x(#{com.join(' ')})
76
+ unless $?.success?
77
+ errstr = "unable to retrieves holds for snapshot: #{snapshot}"
78
+ $logger.error(errstr)
79
+ raise errstr
80
+ end
81
+ a = []
82
+ out.split("\n").each do |ln|
83
+ a.push(ln.split("\t")[1])
84
+ end
85
+ a
86
+ end
87
+
88
+ def self.zfs_hold(hold,snapshot)
89
+ com = [global_options['zfs_binary'], 'hold', hold, snapshot]
90
+ $logger.debug("#{com.join(' ')}")
91
+ system(com.join(' '))
92
+ unless $?.success?
93
+ errstr = "unable to set hold: #{hold} for snapshot: #{snapshot}"
94
+ $logger.error(errstr)
95
+ raise errstr
96
+ end
97
+ end
98
+
99
+ def self.zfs_release(hold,snapshot)
100
+ com = [@global_options['zfs_binary'], 'release', hold, snapshot]
101
+ $logger.debug("#{com.join(' ')}")
102
+ system(com.join(' '))
103
+ unless $?.success?
104
+ errstr = "unable to release hold: #{hold} for snapshot: #{snapshot}"
105
+ $logger.error(errstr)
106
+ raise errstr
107
+ end
108
+ end
109
+
68
110
  def self.zfsget(properties: ['name'],types: ['filesystem','volume'],zfs: '')
69
111
  results={}
70
- com = ['zfs', 'get', '-Hp', properties.join(','), '-t', types.join(','), zfs]
112
+ com = [ZfsMgmt.global_options[:zfs_binary], 'get', '-Hp', properties.join(','), '-t', types.join(','), zfs]
113
+ $logger.debug(com.join(' '))
71
114
  so,se,status = Open3.capture3(com.join(' '))
72
115
  if status.signaled?
73
116
  $logger.error("process was signalled \"#{com.join(' ')}\", termsig #{status.termsig}")
@@ -192,24 +235,25 @@ module ZfsMgmt
192
235
  }
193
236
  return saved,saved_snaps,deleteme
194
237
  end
195
- def self.zfs_managed_list(filter: '.+')
238
+ def self.zfs_managed_list(filter: '.+', properties: custom_properties(), property_match: { 'zfsmgmt:manage' => 'true' } )
196
239
  zfss = [] # array of arrays
197
- zfsget(properties: custom_properties()).each do |zfs,props|
240
+ zfsget(properties: properties).each do |zfs,props|
198
241
  unless /#{filter}/ =~ zfs
199
242
  next
200
243
  end
201
- unless props.has_key?('zfsmgmt:manage') and props['zfsmgmt:manage'] == 'true'
202
- next
244
+ managed = true
245
+ property_match.each do |k,v|
246
+ unless props.has_key?(k) and props[k] == v
247
+ managed = false
248
+ break
249
+ end
203
250
  end
251
+ next unless managed
204
252
  snaps = self.zfsget(properties: ['name','creation','userrefs','used','written','referenced'],types: ['snapshot'], zfs: zfs)
205
253
  if snaps.length == 0
206
254
  $logger.warn("unable to process this zfs, no snapshots at all: #{zfs}")
207
255
  next
208
256
  end
209
- unless props.has_key?('zfsmgmt:policy') and policy = policy_parser(props['zfsmgmt:policy'])
210
- $logger.error("zfs_mgmt is configured to manage #{zfs}, but there is no valid policy configuration, skipping")
211
- next # zfs
212
- end
213
257
  zfss.push([zfs,props,snaps])
214
258
  end
215
259
  return zfss
@@ -222,6 +266,10 @@ module ZfsMgmt
222
266
  end
223
267
  zfs_managed_list(filter: filter).each do |zdata|
224
268
  (zfs,props,snaps) = zdata
269
+ unless props.has_key?('zfsmgmt:policy') and policy_parser(props['zfsmgmt:policy'])
270
+ $logger.error("zfs_mgmt is configured to manage #{zfs}, but there is no valid policy configuration, skipping")
271
+ next # zfs
272
+ end
225
273
  # call the function that decides who to save and who to delete
226
274
  (saved,saved_snaps,deleteme) = snapshot_destroy_policy(zfs,props,snaps)
227
275
 
@@ -247,6 +295,11 @@ module ZfsMgmt
247
295
  end
248
296
  zfs_managed_list(filter: filter).each do |zdata|
249
297
  (zfs,props,snaps) = zdata
298
+ unless props.has_key?('zfsmgmt:policy') and policy_parser(props['zfsmgmt:policy'])
299
+ $logger.error("zfs_mgmt is configured to manage #{zfs}, but there is no valid policy configuration, skipping")
300
+ next # zfs
301
+ end
302
+
250
303
  # call the function that decides who to save and who to delete
251
304
  (saved,saved_snaps,deleteme) = snapshot_destroy_policy(zfs,props,snaps)
252
305
 
@@ -322,7 +375,7 @@ module ZfsMgmt
322
375
 
323
376
  prefix = ( props.has_key?('zfsmgmt:snap_prefix') ? props['zfsmgmt:snap_prefix'] : 'zfsmgmt' )
324
377
  ts = ( props.has_key?('zfsmgmt:snap_timestamp') ? props['zfsmgmt:snap_timestamp'] : '%FT%T%z' )
325
- com = ['zfs','snapshot']
378
+ com = [global_options['zfs_binary'],'snapshot']
326
379
  if props['zfsmgmt:snapshot'] == 'recursive' and props['zfsmgmt:snapshot@source'] == 'local'
327
380
  com.push('-r')
328
381
  end
@@ -333,26 +386,3 @@ module ZfsMgmt
333
386
  end
334
387
  end
335
388
  end
336
- class ZfsMgmtList < Thor
337
- class_option :filter, :type => :string, :default => '.+',
338
- :desc => 'only act on zfs matching this regexp'
339
- desc "stale", "list all zfs with stale snapshots"
340
- method_option :age, :desc => "timeframe outside of which the zfs will be considered stale", :default => '1d'
341
- def stale()
342
- cutoff = Time.at(Time.now.to_i - ZfsMgmt.timespec_to_seconds(options[:age]))
343
- table = Text::Table.new
344
- table.head = ['zfs','snapshot','age']
345
- table.rows = []
346
- ZfsMgmt.zfs_managed_list(filter: options[:filter]).each do |blob|
347
- zfs,props,snaps = blob
348
- last = snaps.keys.sort { |a,b| snaps[a]['creation'] <=> snaps[b]['creation'] }.last
349
- snap_time = Time.at(snaps[last]['creation'])
350
- if snap_time < cutoff
351
- table.rows << [zfs,last.split('@')[1],snap_time]
352
- end
353
- end
354
- if table.rows.count > 0
355
- print table.to_s
356
- end
357
- end
358
- end
@@ -1,33 +1,162 @@
1
+ require "json"
1
2
 
2
3
  module ZfsMgmt::Restic
3
- def self.backup(backup_type: 'incremental', force: false, filter: '.+')
4
- pp backup_type, force, filter
5
- end
6
- end
4
+ def self.restic_snapshots(zfs,options,props)
5
+ # query the restic database
6
+ com = [ options[:restic_binary],
7
+ 'snapshots',
8
+ '--json',
9
+ '--tag', 'zfsmgmt',
10
+ '--path', "/#{zfs}",
11
+ ]
12
+ if options.has_key?('password_file')
13
+ com.push('-p',options['password_file'])
14
+ end
15
+ if options.has_key?('repo')
16
+ com.push('--repo', options['repo'])
17
+ elsif props.has_key?('zfsmgmt:restic_repository')
18
+ com.push( '--repo', props['zfsmgmt:restic_repository'] )
19
+ end
7
20
 
8
- class ZfsMgmtResticBackup < Thor
9
- class_option :filter, :type => :string, :default => '.+',
10
- :desc => 'only act on zfs matching this regexp'
11
- class_option :force, :type => :boolean, :default => false,
12
- :desc => 'force create this backup type, fail if it cannot be forced'
13
- desc "incremental", "perform incremental backup"
14
- def incremental()
15
- ZfsMgmt::Restic.backup(backup_type: 'incremental', force: options[:force])
16
- end
17
- desc "differential", "perform differential backup"
18
- def differential()
19
- ZfsMgmt::Restic.backup(backup_type: 'differential', force: options[:force])
20
- end
21
- desc "full", "perform full backup"
22
- def full()
23
- ZfsMgmt::Restic.backup(backup_type: 'full', force: options[:force])
21
+ $logger.debug("#{com.join(' ')}")
22
+ restic_output = %x(#{com.join(' ')})
23
+ unless $?.success?
24
+ $logger.error("unable to query the restic database")
25
+ raise "unable to query the restic database"
26
+ end
27
+ restic_snapshots = JSON.parse(restic_output)
28
+ restic_snapshot_zfs_snapshot_index = {}
29
+ restic_snapshots.each do |snappy|
30
+ snappy['date_time'] = DateTime.parse(snappy['time'])
31
+ if snappy.has_key?('tags')
32
+ snappy['tags'].each do |t|
33
+ if m = /^(zfsmgmt:.+?)=(.+)/.match(t)
34
+ if ['zfsmgmt:level'].include?(m[1])
35
+ snappy[m[1]] = m[2].to_i
36
+ else
37
+ snappy[m[1]] = m[2]
38
+ end
39
+ if m[1] == 'zfsmgmt:snapshot'
40
+ restic_snapshot_zfs_snapshot_index[m[2]] = snappy
41
+ end
42
+ end
43
+ end
44
+ end
45
+ end
46
+ return([restic_snapshots,restic_snapshot_zfs_snapshot_index])
24
47
  end
25
- end
26
48
 
27
- class ZfsMgmtRestic < Thor
28
- desc "restic SUBCOMMAND ...ARGS", "backup all configured zfs to restic"
29
- subcommand "backup", ZfsMgmtResticBackup
49
+ def self.valid_chain(snap,restic_snapshots,restic_snapshot_zfs_snapshot_index,a)
50
+ if snap['zfsmgmt:level'] == 0
51
+ a.push(snap)
52
+ $logger.debug("found complete chain culminating in full backup of: #{snap['zfsmgmt:snapshot']}")
53
+ return a
54
+ elsif restic_snapshot_zfs_snapshot_index.has_key?(snap['zfsmgmt:parent'])
55
+ a.push(snap)
56
+ $logger.debug("found another link in the chain: #{snap['zfsmgmt:snapshot']} => #{snap['zfsmgmt:parent']}")
57
+ return valid_chain(restic_snapshot_zfs_snapshot_index[snap['zfsmgmt:parent']],restic_snapshots,restic_snapshot_zfs_snapshot_index,a)
58
+ else
59
+ $logger.error("broken chain: looking for the parent of #{snap['zfsmgmt:snapshot']} (#{snap['zfsmgmt:parent']}) and failed to find")
60
+ return []
61
+ end
62
+ end
63
+
64
+
65
+ def self.backup(backup_level: 2,
66
+ options: {})
67
+ ZfsMgmt.zfs_managed_list(filter: options['filter'],
68
+ properties: ['name',
69
+ 'zfsmgmt:restic_backup',
70
+ 'zfsmgmt:restic_repository',
71
+ 'userrefs',
72
+ ],
73
+ property_match: { 'zfsmgmt:restic_backup' => 'true' }).each do |blob|
74
+ zfs,props,zfs_snapshots = blob
75
+ last_zfs_snapshot = zfs_snapshots.keys.sort { |a,b| zfs_snapshots[a]['creation'] <=> zfs_snapshots[b]['creation'] }.last
76
+ zfs_snap_time = Time.at(zfs_snapshots[last_zfs_snapshot]['creation'])
30
77
 
78
+ level = 0
79
+ chain = []
80
+ zfs_snap_parent = ''
81
+ restic_snap_parent = ''
82
+ (restic_snapshots,restic_snapshot_zfs_snapshot_index) = restic_snapshots(zfs,options,props)
83
+ if restic_snapshot_zfs_snapshot_index.has_key?(last_zfs_snapshot)
84
+ $logger.warn("backup of this snapshot #{last_zfs_snapshot} already exists in restic, cannot continue with backup of #{zfs}")
85
+ next # next zfs filesystem to be backed up
86
+ end
87
+ if backup_level > 0 and restic_snapshots.count > 0
88
+ # reverse (oldest first) sorted restic snapshots
89
+ restic_snap_parent = restic_snapshots.filter { |rsnap|
90
+ rsnap.has_key?('zfsmgmt:zfs') and rsnap['zfsmgmt:zfs'] == zfs and
91
+ rsnap.has_key?('zfsmgmt:level') and rsnap['zfsmgmt:level'] < backup_level }.sort {
92
+ |a,b| a['date_time'] <=> b['date_time'] }.last
93
+ if restic_snap_parent and
94
+ zfs_snapshots.has_key?(restic_snap_parent['zfsmgmt:snapshot']) and
95
+ chain = valid_chain(restic_snap_parent,restic_snapshots,restic_snapshot_zfs_snapshot_index,[]) and
96
+ chain.length > 0
97
+
98
+ level = restic_snap_parent['zfsmgmt:level'] + 1
99
+ zfs_snap_parent = restic_snap_parent['zfsmgmt:snapshot']
100
+ $logger.debug("restic_snap_parent: level: #{restic_snap_parent['zfsmgmt:level']} snapshot: #{zfs_snap_parent}")
101
+ else
102
+ $logger.error("restic_snap_parent rejected: level: #{restic_snap_parent['zfsmgmt:level']} snapshot: #{restic_snap_parent['zfsmgmt:snapshot']}")
103
+ end
104
+ $logger.debug("chain of snapshots: #{chain}")
105
+ end
106
+ tags = [ 'zfsmgmt',
107
+ "zfsmgmt:snapshot=#{last_zfs_snapshot}",
108
+ "zfsmgmt:zfs=#{zfs}",
109
+ "zfsmgmt:level=#{level}" ]
110
+ com = [ ZfsMgmt.global_options['zfs_binary'], 'send', '-L', '-w', '-h', '-p' ]
111
+ if level > 0
112
+ if options[:intermediary]
113
+ com.push('-I')
114
+ else
115
+ com.push('-i')
116
+ end
117
+ com.push(zfs_snap_parent)
118
+ tags.push("zfsmgmt:parent=#{zfs_snap_parent}")
119
+ end
120
+ com.push( last_zfs_snapshot )
121
+ com.push( '|', 'mbuffer', '-m', options[:buffer], '-q' )
122
+ com.push( '|', options[:restic_binary], 'backup', '--stdin',
123
+ '--stdin-filename', zfs, '--time', "\"#{zfs_snap_time.strftime('%F %T')}\"" )
124
+ tags.each do |tag|
125
+ com.push( '--tag', "\"#{tag}\"" )
126
+ end
127
+ if options.has_key?('limit_upload')
128
+ com.push('--limit-upload', options['limit_upload'])
129
+ end
130
+ if options.has_key?('password_file')
131
+ com.push('-p',options['password_file'])
132
+ end
133
+ if options.has_key?('repo')
134
+ com.push('--repo', options['repo'])
135
+ elsif props.has_key?('zfsmgmt:restic_repository')
136
+ com.push( '--repo', props['zfsmgmt:restic_repository'] )
137
+ end
138
+ if options[:verbose]
139
+ com.push('--verbose',options[:verbose])
140
+ elsif $stdout.isatty
141
+ com.push('-v')
142
+ end
143
+ unless ZfsMgmt.zfs_holds(last_zfs_snapshot).include?('zfsmgmt_restic')
144
+ ZfsMgmt.zfs_hold('zfsmgmt_restic',last_zfs_snapshot)
145
+ end
146
+ $logger.info("#{com.join(' ')}")
147
+ system(com.join(' '))
148
+ chain_snaps = chain.map do |rsnap|
149
+ rsnap['zfsmgmt:snapshot']
150
+ end
151
+ zfs_snapshots.each do |s,d|
152
+ d['userrefs'] == 0 and next
153
+ chain_snaps.include?(s) and next
154
+ s == last_zfs_snapshot and next
155
+ if ZfsMgmt.zfs_holds(s).include?('zfsmgmt_restic')
156
+ ZfsMgmt.zfs_release('zfsmgmt_restic',s)
157
+ end
158
+ end
159
+ end
160
+ end
31
161
  end
32
162
 
33
-
@@ -1,3 +1,3 @@
1
1
  module ZfsMgmt
2
- VERSION = "0.3.5"
2
+ VERSION = "0.3.10"
3
3
  end
@@ -0,0 +1,6 @@
1
+
2
+ module ZfsMgmt::ZfsMgr
3
+ require "zfs_mgmt/zfs_mgr/list"
4
+ require "zfs_mgmt/zfs_mgr/restic"
5
+ require "zfs_mgmt/zfs_mgr/snapshot"
6
+ end
@@ -0,0 +1,26 @@
1
+ # class list
2
+
3
+ class ZfsMgmt::ZfsMgr::List < Thor
4
+ class_option :filter, :type => :string, :default => '.+',
5
+ :desc => 'only act on zfs matching this regexp'
6
+ desc "stale", "list all zfs with stale snapshots"
7
+ method_option :age, :desc => "timeframe outside of which the zfs will be considered stale", :default => '1d'
8
+ def stale()
9
+ ZfsMgmt.global_options = options
10
+ cutoff = Time.at(Time.now.to_i - ZfsMgmt.timespec_to_seconds(options[:age]))
11
+ table = Text::Table.new
12
+ table.head = ['zfs','snapshot','age']
13
+ table.rows = []
14
+ ZfsMgmt.zfs_managed_list(filter: options[:filter]).each do |blob|
15
+ zfs,props,snaps = blob
16
+ last = snaps.keys.sort { |a,b| snaps[a]['creation'] <=> snaps[b]['creation'] }.last
17
+ snap_time = Time.at(snaps[last]['creation'])
18
+ if snap_time < cutoff
19
+ table.rows << [zfs,last.split('@')[1],snap_time]
20
+ end
21
+ end
22
+ if table.rows.count > 0
23
+ print table.to_s
24
+ end
25
+ end
26
+ end
@@ -0,0 +1,41 @@
1
+
2
+ class ZfsMgmt::ZfsMgr::Backup < Thor
3
+ include ZfsMgmt::Restic
4
+ class_option :filter, :type => :string, :default => '.+',
5
+ :desc => 'only act on zfs matching this regexp'
6
+ class_option :restic_binary, :type => :string, :default => 'restic',
7
+ :desc => 'restic binary'
8
+ class_option :verbose, :alias => '-v', :type => :numeric,
9
+ :desc => 'verbosity level for restic'
10
+ class_option :buffer, :type => :string, :default => '256m',
11
+ :desc => 'buffer size for mbuffer'
12
+ class_option :password_file, :alias => '-p', :type => :string,
13
+ :desc => 'passed to restic'
14
+ class_option :limit_upload, :type => :numeric,
15
+ :desc => 'passed to restic'
16
+ class_option :repo, :type => :string,
17
+ :desc => 'passed to restic'
18
+ desc "incremental", "perform incremental backup"
19
+ method_option :level, :desc => "backup level in integer form", :default => 2, :type => :numeric
20
+ method_option :intermediary, :alias => '-I', :desc => "pass -I (intermediary) option to zfs send", :default => false, :type => :boolean
21
+ def incremental()
22
+ ZfsMgmt.global_options = options
23
+ ZfsMgmt::Restic.backup(backup_level: options[:level], options: options)
24
+ end
25
+ desc "differential", "perform differential backup"
26
+ method_option :intermediary, :alias => '-I', :desc => "pass -I (intermediary) option to zfs send", :default => false, :type => :boolean
27
+ def differential()
28
+ ZfsMgmt.global_options = options
29
+ ZfsMgmt::Restic.backup(backup_level: 1, options: options)
30
+ end
31
+ desc "full", "perform full backup"
32
+ def full()
33
+ ZfsMgmt.global_options = options
34
+ ZfsMgmt::Restic.backup(backup_level: 0, options: options)
35
+ end
36
+ end
37
+
38
+ class ZfsMgmt::ZfsMgr::Restic < Thor
39
+ desc "backup SUBCOMMAND ...ARGS", "backup all configured zfs to restic"
40
+ subcommand "backup", ZfsMgmt::ZfsMgr::Backup
41
+ end
@@ -0,0 +1,27 @@
1
+ # implement snapshot management
2
+
3
+ class ZfsMgmt::ZfsMgr::Snapshot < Thor
4
+ class_option :noop, :type => :boolean, :default => false,
5
+ :desc => 'pass -n option to zfs commands'
6
+ class_option :verbose, :type => :boolean, :default => false,
7
+ :desc => 'pass -v option to zfs commands'
8
+ class_option :debug, :type => :boolean, :default => false,
9
+ :desc => 'set logging level to debug'
10
+ class_option :filter, :type => :string, :default => '.+',
11
+ :desc => 'only act on zfs matching this regexp'
12
+ desc "destroy", "apply the snapshot destroy policy to zfs"
13
+ def destroy()
14
+ ZfsMgmt.global_options = options
15
+ ZfsMgmt.snapshot_destroy(noop: options[:noop], verbopt: options[:verbose], debugopt: options[:debug], filter: options[:filter])
16
+ end
17
+ desc "policy", "print the policy table for zfs"
18
+ def policy()
19
+ ZfsMgmt.global_options = options
20
+ ZfsMgmt.snapshot_policy(verbopt: options[:verbose], debugopt: options[:debug], filter: options[:filter])
21
+ end
22
+ desc "create", "execute zfs snapshot based on zfs properties"
23
+ def create()
24
+ ZfsMgmt.global_options = options
25
+ ZfsMgmt.snapshot_create(verbopt: options[:verbose], debugopt: options[:debug], filter: options[:filter])
26
+ end
27
+ end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: zfs_mgmt
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.3.5
4
+ version: 0.3.10
5
5
  platform: ruby
6
6
  authors:
7
7
  - Aran Cox
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2021-02-13 00:00:00.000000000 Z
11
+ date: 2021-02-21 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: bundler
@@ -127,6 +127,10 @@ files:
127
127
  - lib/zfs_mgmt.rb
128
128
  - lib/zfs_mgmt/restic.rb
129
129
  - lib/zfs_mgmt/version.rb
130
+ - lib/zfs_mgmt/zfs_mgr.rb
131
+ - lib/zfs_mgmt/zfs_mgr/list.rb
132
+ - lib/zfs_mgmt/zfs_mgr/restic.rb
133
+ - lib/zfs_mgmt/zfs_mgr/snapshot.rb
130
134
  - zfs_mgmt.gemspec
131
135
  homepage: https://github.com/aranc23/zfs_mgmt
132
136
  licenses: