backup_zh 4.0.3.1 → 4.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (53) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +15 -7
  3. data/lib/backup.rb +6 -0
  4. data/lib/backup/cli.rb +10 -0
  5. data/lib/backup/config/dsl.rb +3 -3
  6. data/lib/backup/database/mysql.rb +17 -6
  7. data/lib/backup/database/postgresql.rb +2 -2
  8. data/lib/backup/database/sqlite.rb +57 -0
  9. data/lib/backup/encryptor/open_ssl.rb +7 -2
  10. data/lib/backup/model.rb +26 -1
  11. data/lib/backup/notifier/base.rb +30 -0
  12. data/lib/backup/notifier/campfire.rb +1 -7
  13. data/lib/backup/notifier/command.rb +99 -0
  14. data/lib/backup/notifier/datadog.rb +107 -0
  15. data/lib/backup/notifier/flowdock.rb +6 -5
  16. data/lib/backup/notifier/hipchat.rb +27 -8
  17. data/lib/backup/notifier/http_post.rb +2 -7
  18. data/lib/backup/notifier/mail.rb +19 -10
  19. data/lib/backup/notifier/nagios.rb +3 -8
  20. data/lib/backup/notifier/pagerduty.rb +81 -0
  21. data/lib/backup/notifier/prowl.rb +8 -9
  22. data/lib/backup/notifier/pushover.rb +1 -7
  23. data/lib/backup/notifier/ses.rb +88 -0
  24. data/lib/backup/notifier/slack.rb +7 -17
  25. data/lib/backup/notifier/twitter.rb +1 -7
  26. data/lib/backup/notifier/zabbix.rb +1 -6
  27. data/lib/backup/package.rb +4 -0
  28. data/lib/backup/packager.rb +8 -2
  29. data/lib/backup/storage/base.rb +15 -3
  30. data/lib/backup/storage/cycler.rb +24 -14
  31. data/lib/backup/storage/ftp.rb +15 -1
  32. data/lib/backup/storage/s3.rb +2 -1
  33. data/lib/backup/syncer/base.rb +2 -2
  34. data/lib/backup/version.rb +1 -1
  35. data/templates/cli/config +2 -2
  36. data/templates/cli/databases/sqlite +11 -0
  37. data/templates/cli/model +1 -1
  38. data/templates/cli/notifiers/command +32 -0
  39. data/templates/cli/notifiers/datadog +57 -0
  40. data/templates/cli/notifiers/hipchat +1 -0
  41. data/templates/cli/notifiers/mail +3 -0
  42. data/templates/cli/notifiers/pagerduty +12 -0
  43. data/templates/cli/notifiers/ses +15 -0
  44. data/templates/cli/notifiers/slack +3 -4
  45. data/templates/cli/storages/dropbox +1 -0
  46. data/templates/cli/storages/ftp +1 -0
  47. data/templates/cli/storages/local +1 -0
  48. data/templates/cli/storages/ninefold +1 -0
  49. data/templates/cli/storages/s3 +2 -0
  50. data/templates/cli/storages/scp +1 -0
  51. data/templates/cli/storages/sftp +1 -0
  52. data/templates/general/links +3 -3
  53. metadata +53 -136
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 2a5011f86294da25fd2e264a84a51bce114bcc8a
4
- data.tar.gz: 4b0f2ae51ead1af4a112028233cfff9f6c1fe8e4
3
+ metadata.gz: 7fc3367bd1a9cebd38de0ac0c15c1b81dfaa3d2f
4
+ data.tar.gz: 986424c3cce771ff3e7dad33d6fa72437381387f
5
5
  SHA512:
6
- metadata.gz: 496a87045fd14afc4010458132c8d7a38306a5952abb0781f1ce2e5f54d49eba9680f12e974eabae8ac3cac86a9d75a620f076e634439d7e1eecb822a9dda555
7
- data.tar.gz: c1e4e9403cfca3b4fc228ee0f6cdbeba389836042e2c7efb527174792b09305a56fdef0d8491ee311174ff11be9ae495cfbba1476401a5da73281cf18d179621
6
+ metadata.gz: 94aed25187160ef0525c37f22f8ddef4c0afbe54e60103d21e54446a60a1c1277837045ece5ffbf6d7270aba4149a7a040588512957ac50b5fa62f807e37850f
7
+ data.tar.gz: 34a46c8e8ad9d11611af22671ac1979f11a9c005baca98cfc56358c5a815b5ccf4d964afb5ecebc360f3f7ffa6ada951762564b6599b7e46256a12336e435585
data/README.md CHANGED
@@ -1,21 +1,29 @@
1
1
  Backup v4.x
2
2
  ===========
3
- [![Code Climate](https://codeclimate.com/github/meskyanichi/backup.png)](https://codeclimate.com/github/meskyanichi/backup)
4
- [![Build Status](https://travis-ci.org/meskyanichi/backup.svg?branch=master)](https://travis-ci.org/meskyanichi/backup)
3
+
4
+ [![Code Climate](https://codeclimate.com/github/backup/backup.png)](https://codeclimate.com/github/backup/backup)
5
+ [![Build Status](https://travis-ci.org/backup/backup.svg?branch=master)](https://travis-ci.org/backup/backup)
6
+ [![Join the chat at https://gitter.im/backup/backup](https://badges.gitter.im/Join%20Chat.svg)][Gitter]
5
7
 
6
8
  Backup is a system utility for Linux and Mac OS X, distributed as a RubyGem, that allows you to easily perform backup
7
9
  operations. It provides an elegant DSL in Ruby for _modeling_ your backups. Backup has built-in support for various
8
10
  databases, storage protocols/services, syncers, compressors, encryptors and notifiers which you can mix and match. It
9
11
  was built with modularity, extensibility and simplicity in mind.
10
12
 
11
- [Installation][] · [Release Notes][] · [Documentation][]
13
+ [Installation][] · [Release Notes][] · [Documentation][] · [Issues][] · [Features][] · [Chat][Gitter]
12
14
 
15
+ Please use the Backup features [issue tracker][Features] to suggest new features.
16
+ Only use the Backup gem [issue tracker][Issues] for bugs and other issues.
17
+ We're also available on [Gitter][] for questions and problems.
13
18
 
14
- **Copyright (c) 2009-2014 [Michael van Rooijen][] ( [@meskyanichi][] )**
19
+ **Copyright (c) 2009-2015 [Michael van Rooijen][] ( [@meskyanichi][] )**
15
20
  Released under the **MIT** [License](LICENSE.md).
16
21
 
17
- [Installation]: http://meskyanichi.github.io/backup/v4/installation
18
- [Release Notes]: http://meskyanichi.github.io/backup/v4/release-notes
19
- [Documentation]: http://meskyanichi.github.io/backup/v4
22
+ [Installation]: http://backup.github.io/backup/v4/installation
23
+ [Release Notes]: http://backup.github.io/backup/v4/release-notes
24
+ [Documentation]: http://backup.github.io/backup/v4
25
+ [Issues]: https://github.com/backup/backup/issues
26
+ [Features]: https://github.com/backup/backup-features/issues
27
+ [Gitter]: https://gitter.im/backup/backup?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge
20
28
  [Michael van Rooijen]: http://michaelvanrooijen.com
21
29
  [@meskyanichi]: http://twitter.com/#!/meskyanichi
@@ -11,6 +11,7 @@ require 'thread'
11
11
 
12
12
  require 'open4'
13
13
  require 'thor'
14
+ require 'shellwords'
14
15
 
15
16
  require 'excon'
16
17
  # Include response.inspect in error messages.
@@ -78,6 +79,7 @@ module Backup
78
79
  autoload :Redis, File.join(DATABASE_PATH, 'redis')
79
80
  autoload :Riak, File.join(DATABASE_PATH, 'riak')
80
81
  autoload :OpenLDAP, File.join(DATABASE_PATH, 'openldap')
82
+ autoload :SQLite, File.join(DATABASE_PATH, 'sqlite')
81
83
  end
82
84
 
83
85
  ##
@@ -106,12 +108,16 @@ module Backup
106
108
  autoload :Campfire, File.join(NOTIFIER_PATH, 'campfire')
107
109
  autoload :Prowl, File.join(NOTIFIER_PATH, 'prowl')
108
110
  autoload :Hipchat, File.join(NOTIFIER_PATH, 'hipchat')
111
+ autoload :PagerDuty, File.join(NOTIFIER_PATH, 'pagerduty')
109
112
  autoload :Pushover, File.join(NOTIFIER_PATH, 'pushover')
110
113
  autoload :Slack, File.join(NOTIFIER_PATH, 'slack')
111
114
  autoload :HttpPost, File.join(NOTIFIER_PATH, 'http_post')
112
115
  autoload :Nagios, File.join(NOTIFIER_PATH, 'nagios')
113
116
  autoload :FlowDock, File.join(NOTIFIER_PATH, 'flowdock')
114
117
  autoload :Zabbix, File.join(NOTIFIER_PATH, 'zabbix')
118
+ autoload :DataDog, File.join(NOTIFIER_PATH, 'datadog')
119
+ autoload :Ses, File.join(NOTIFIER_PATH, 'ses')
120
+ autoload :Command, File.join(NOTIFIER_PATH, 'command')
115
121
  end
116
122
 
117
123
  ##
@@ -142,6 +142,9 @@ module Backup
142
142
 
143
143
  rescue Exception => err
144
144
  Logger.error Error.wrap(err)
145
+ unless Helpers.is_backup_error? err
146
+ Logger.error err.backtrace.join("\n")
147
+ end
145
148
  # Logger configuration will be ignored
146
149
  # and messages will be output to the console only.
147
150
  Logger.abort!
@@ -219,6 +222,9 @@ module Backup
219
222
  Config.load(options)
220
223
  rescue Exception => err
221
224
  Logger.error Error.wrap(err)
225
+ unless Helpers.is_backup_error? err
226
+ Logger.error err.backtrace.join("\n")
227
+ end
222
228
  end
223
229
 
224
230
  if Logger.has_warnings? || Logger.has_errors?
@@ -357,6 +363,10 @@ module Backup
357
363
  exec(cmd)
358
364
  end
359
365
 
366
+ def is_backup_error?(error)
367
+ error.class.ancestors.include? Backup::Error
368
+ end
369
+
360
370
  end
361
371
  end
362
372
 
@@ -27,7 +27,7 @@ module Backup
27
27
  create_modules(
28
28
  DSL,
29
29
  [ # Databases
30
- ['MySQL', 'PostgreSQL', 'MongoDB', 'Redis', 'Riak', 'OpenLDAP'],
30
+ ['MySQL', 'PostgreSQL', 'MongoDB', 'Redis', 'Riak', 'OpenLDAP', 'SQLite'],
31
31
  # Storages
32
32
  ['S3', 'CloudFiles', 'Ninefold', 'Dropbox', 'FTP',
33
33
  'SFTP', 'SCP', 'RSync', 'Local', 'QiNiu'],
@@ -42,8 +42,8 @@ module Backup
42
42
  ],
43
43
  # Notifiers
44
44
  ['Mail', 'Twitter', 'Campfire', 'Prowl',
45
- 'Hipchat', 'Pushover', 'HttpPost', 'Nagios',
46
- 'Slack', 'FlowDock', 'Zabbix']
45
+ 'Hipchat', 'PagerDuty', 'Pushover', 'HttpPost', 'Nagios',
46
+ 'Slack', 'FlowDock', 'Zabbix', 'Ses', 'DataDog', 'Command']
47
47
  ]
48
48
  )
49
49
  end
@@ -1,5 +1,4 @@
1
1
  # encoding: utf-8
2
- require 'shellwords'
3
2
 
4
3
  module Backup
5
4
  module Database
@@ -47,6 +46,12 @@ module Backup
47
46
  # See: http://www.percona.com/doc/percona-xtrabackup/
48
47
  attr_accessor :backup_engine
49
48
 
49
+ ##
50
+ # If true (which is the default behaviour), the backup will be prepared
51
+ # after it has been successfuly created. This option is only valid if
52
+ # :backup_engine is set to :innobackupex.
53
+ attr_accessor :prepare_backup
54
+
50
55
  ##
51
56
  # If set the backup engine command block is executed as the given user
52
57
  attr_accessor :sudo_user
@@ -61,6 +66,7 @@ module Backup
61
66
 
62
67
  @name ||= :all
63
68
  @backup_engine ||= :mysqldump
69
+ @prepare_backup = true if @prepare_backup.nil?
64
70
  end
65
71
 
66
72
  ##
@@ -95,8 +101,8 @@ module Backup
95
101
  private
96
102
 
97
103
  def mysqldump
98
- "#{ utility(:mysqldump) } #{ credential_options } " +
99
- "#{ connectivity_options } #{ user_options } #{ name_option } " +
104
+ "#{ utility(:mysqldump) } #{ user_options } #{ credential_options } " +
105
+ "#{ connectivity_options } #{ name_option } " +
100
106
  "#{ tables_to_dump } #{ tables_to_skip }"
101
107
  end
102
108
 
@@ -152,14 +158,19 @@ module Backup
152
158
  "#{ utility(:innobackupex) } #{ credential_options } " +
153
159
  "#{ connectivity_options } #{ user_options } " +
154
160
  "--no-timestamp #{ temp_dir } #{ quiet_option } && " +
155
- # Log applying phase (prepare for restore)
156
- "#{ utility(:innobackupex) } --apply-log #{ temp_dir } " +
157
- "#{ user_prepare_options } #{ quiet_option } && " +
161
+ innobackupex_prepare +
158
162
  # Move files to tar-ed stream on stdout
159
163
  "#{ utility(:tar) } --remove-files -cf - " +
160
164
  "-C #{ File.dirname(temp_dir) } #{ File.basename(temp_dir) }"
161
165
  end
162
166
 
167
+ def innobackupex_prepare
168
+ return "" unless @prepare_backup
169
+ # Log applying phase (prepare for restore)
170
+ "#{ utility(:innobackupex) } --apply-log #{ temp_dir } " +
171
+ "#{ user_prepare_options } #{ quiet_option } && "
172
+ end
173
+
163
174
  def sudo_option(command_block)
164
175
  return command_block unless sudo_user
165
176
 
@@ -88,7 +88,7 @@ module Backup
88
88
  end
89
89
 
90
90
  def password_option
91
- "PGPASSWORD='#{ password }' " if password
91
+ "PGPASSWORD=#{ Shellwords.escape(password) } " if password
92
92
  end
93
93
 
94
94
  def sudo_option
@@ -96,7 +96,7 @@ module Backup
96
96
  end
97
97
 
98
98
  def username_option
99
- "--username='#{ username }'" if username
99
+ "--username=#{ Shellwords.escape(username) }" if username
100
100
  end
101
101
 
102
102
  def connectivity_options
@@ -0,0 +1,57 @@
1
+ # encoding: utf-8
2
+
3
+ module Backup
4
+ module Database
5
+ class SQLite < Base
6
+ class Error < Backup::Error; end
7
+
8
+ ##
9
+ # Path to the sqlite3 file
10
+ attr_accessor :path
11
+
12
+ ##
13
+ # Path to sqlite utility (optional)
14
+ attr_accessor :sqlitedump_utility
15
+
16
+ ##
17
+ # Creates a new instance of the SQLite adapter object
18
+ def initialize(model, database_id = nil, &block)
19
+ super
20
+ instance_eval(&block) if block_given?
21
+
22
+ @sqlitedump_utility ||= utility(:sqlitedump)
23
+ end
24
+
25
+ ##
26
+ # Performs the sqlitedump command and outputs the
27
+ # data to the specified path based on the 'trigger'
28
+ def perform!
29
+ super
30
+
31
+ dump = "echo '.dump' | #{ sqlitedump_utility } #{ path }"
32
+
33
+ pipeline = Pipeline.new
34
+ dump_ext = 'sql'
35
+
36
+ pipeline << dump
37
+ if model.compressor
38
+ model.compressor.compress_with do |command, ext|
39
+ pipeline << command
40
+ dump_ext << ext
41
+ end
42
+ end
43
+
44
+ pipeline << "cat > '#{ File.join( dump_path , dump_filename) }.#{ dump_ext }'"
45
+
46
+ pipeline.run
47
+
48
+ if pipeline.success?
49
+ log!(:finished)
50
+ else
51
+ raise Error,
52
+ "#{ database_name } Dump Failed!\n" + pipeline.error_messages
53
+ end
54
+ end
55
+ end
56
+ end
57
+ end
@@ -62,8 +62,13 @@ module Backup
62
62
  opts = ['aes-256-cbc']
63
63
  opts << '-base64' if @base64
64
64
  opts << '-salt' if @salt
65
- opts << ( @password_file.to_s.empty? ?
66
- "-k '#{@password}'" : "-pass file:#{@password_file}" )
65
+
66
+ if @password_file.to_s.empty?
67
+ opts << "-k #{Shellwords.escape(@password)}"
68
+ else
69
+ opts << "-pass file:#{@password_file}"
70
+ end
71
+
67
72
  opts.join(' ')
68
73
  end
69
74
 
@@ -303,7 +303,7 @@ module Backup
303
303
  return [] unless databases.any? || archives.any?
304
304
 
305
305
  [lambda { prepare! }, databases, archives,
306
- lambda { package! }, storages, lambda { clean! }]
306
+ lambda { package! }, lambda { store! }, lambda { clean! }]
307
307
  end
308
308
 
309
309
  ##
@@ -325,6 +325,31 @@ module Backup
325
325
  Cleaner.remove_packaging(self)
326
326
  end
327
327
 
328
+ ##
329
+ # Attempts to use all configured Storages, even if some of them result in exceptions.
330
+ # Returns true or raises first encountered exception.
331
+ def store!
332
+ storage_results = storages.map do |storage|
333
+ begin
334
+ storage.perform!
335
+ rescue => ex
336
+ ex
337
+ end
338
+ end
339
+
340
+ first_exception, *other_exceptions = storage_results.select { |result| result.is_a? Exception }
341
+
342
+ if first_exception
343
+ other_exceptions.each do |exception|
344
+ Logger.error exception.to_s
345
+ Logger.error exception.backtrace.join('\n')
346
+ end
347
+ raise first_exception
348
+ else
349
+ true
350
+ end
351
+ end
352
+
328
353
  ##
329
354
  # Removes the final package file(s) once all configured Storages have run.
330
355
  def clean!
@@ -36,6 +36,15 @@ module Backup
36
36
  # Default: 30
37
37
  attr_accessor :retry_waitsec
38
38
 
39
+ ##
40
+ # Message to send. Depends on notifier implementation if this is used.
41
+ # Default: lambda returning:
42
+ # "#{ message } #{ model.label } (#{ model.trigger })"
43
+ #
44
+ # @yieldparam [model] Backup::Model
45
+ # @yieldparam [data] Hash containing `message` and `key` values.
46
+ attr_accessor :message
47
+
39
48
  attr_reader :model
40
49
 
41
50
  def initialize(model)
@@ -47,6 +56,9 @@ module Backup
47
56
  @on_failure = true if on_failure.nil?
48
57
  @max_retries ||= 10
49
58
  @retry_waitsec ||= 30
59
+ @message ||= lambda do |model, data|
60
+ "[#{ data[:status][:message] }] #{ model.label } (#{ model.trigger })"
61
+ end
50
62
  end
51
63
 
52
64
  # This method is called from an ensure block in Model#perform! and must
@@ -93,6 +105,24 @@ module Backup
93
105
  self.class.to_s.sub('Backup::', '')
94
106
  end
95
107
 
108
+ ##
109
+ # Return status data for message creation
110
+ def status_data_for(status)
111
+ {
112
+ :success => {
113
+ :message => 'Backup::Success',
114
+ :key => :success
115
+ },
116
+ :warning => {
117
+ :message => 'Backup::Warning',
118
+ :key => :warning
119
+ },
120
+ :failure => {
121
+ :message => 'Backup::Failure',
122
+ :key => :failure
123
+ }
124
+ }[status]
125
+ end
96
126
  end
97
127
  end
98
128
  end
@@ -42,13 +42,7 @@ module Backup
42
42
  # : Notification will be sent if `on_warning` or `on_success` is `true`.
43
43
  #
44
44
  def notify!(status)
45
- tag = case status
46
- when :success then '[Backup::Success]'
47
- when :warning then '[Backup::Warning]'
48
- when :failure then '[Backup::Failure]'
49
- end
50
- message = "#{ tag } #{ model.label } (#{ model.trigger })"
51
- send_message(message)
45
+ send_message(message.call(model, :status => status_data_for(status)))
52
46
  end
53
47
 
54
48
  def send_message(message)
@@ -0,0 +1,99 @@
1
+ # encoding: utf-8
2
+
3
+ module Backup
4
+ module Notifier
5
+ class Command < Base
6
+
7
+ ##
8
+ # Command to execute.
9
+ #
10
+ # Make sure it is accessible from your $PATH, or provide
11
+ # the absolute path to the command.
12
+ attr_accessor :command
13
+
14
+ ##
15
+ # Arguments to pass to the command.
16
+ #
17
+ # Must be an array of strings or callable objects.
18
+ #
19
+ # Callables will be invoked with #call(model, status),
20
+ # and the return value used as the argument.
21
+ #
22
+ # In strings you can use the following placeholders:
23
+ #
24
+ # %l - Model label
25
+ # %t - Model trigger
26
+ # %s - Status (success/failure/warning)
27
+ # %v - Status verb (succeeded/failed/succeeded with warnings)
28
+ #
29
+ # All placeholders can be used with uppercase letters to capitalize
30
+ # the value.
31
+ #
32
+ # Defaults to ["%L %v"]
33
+ attr_accessor :args
34
+
35
+ def initialize(model, &block)
36
+ super
37
+ instance_eval(&block) if block_given?
38
+
39
+ @args ||= ["%L %v"]
40
+ end
41
+
42
+ private
43
+
44
+ ##
45
+ # Notify the user of the backup operation results.
46
+ #
47
+ # `status` indicates one of the following:
48
+ #
49
+ # `:success`
50
+ # : The backup completed successfully.
51
+ # : Notification will be sent if `on_success` is `true`.
52
+ #
53
+ # `:warning`
54
+ # : The backup completed successfully, but warnings were logged.
55
+ # : Notification will be sent if `on_warning` or `on_success` is `true`.
56
+ #
57
+ # `:failure`
58
+ # : The backup operation failed.
59
+ # : Notification will be sent if `on_warning` or `on_success` is `true`.
60
+ #
61
+ def notify!(status)
62
+ IO.popen([@command] + args.map { |arg| format_arg(arg, status) })
63
+ end
64
+
65
+ def format_arg(arg, status)
66
+ if arg.respond_to?(:call)
67
+ arg.call(model, status)
68
+ else
69
+ arg.gsub(/%(\w)/) do |match|
70
+ ph = match[1]
71
+ val = case ph.downcase
72
+ when "l"
73
+ model.label
74
+ when "t"
75
+ model.trigger.to_s
76
+ when "v"
77
+ status_verb(status)
78
+ when "s"
79
+ status.to_s
80
+ end
81
+ val.capitalize! if ph == ph.upcase
82
+ val
83
+ end
84
+ end
85
+ end
86
+
87
+ def status_verb(status)
88
+ case status
89
+ when :success
90
+ "succeeded"
91
+ when :failure
92
+ "failed"
93
+ when :warning
94
+ "succeeded with warnings"
95
+ end
96
+ end
97
+ end
98
+ end
99
+ end