boring-backup 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +44 -0
- data/LICENSE.txt +21 -0
- data/README.md +115 -0
- data/Rakefile +8 -0
- data/exe/bb +6 -0
- data/lib/boring-backup.rb +1 -0
- data/lib/boring_backup/cli.rb +17 -0
- data/lib/boring_backup/commands/backup.rb +135 -0
- data/lib/boring_backup/configuration.rb +105 -0
- data/lib/boring_backup/jobs/backup_job.rb +11 -0
- data/lib/boring_backup/notifiers/sentinel.rb +103 -0
- data/lib/boring_backup/notifiers/slack.rb +34 -0
- data/lib/boring_backup/notifiers/stdout.rb +7 -0
- data/lib/boring_backup/plugins/rails.rb +9 -0
- data/lib/boring_backup/stores/result.rb +11 -0
- data/lib/boring_backup/stores/s3.rb +119 -0
- data/lib/boring_backup/stores/store.rb +9 -0
- data/lib/boring_backup/stores.rb +3 -0
- data/lib/boring_backup/tee.rb +33 -0
- data/lib/boring_backup/utils.rb +14 -0
- data/lib/boring_backup/version.rb +3 -0
- data/lib/boring_backup.rb +68 -0
- metadata +97 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: 520f4d8e05984b490027c72e0922660b93f73ed22f6d48671ba87b6c4922883e
|
|
4
|
+
data.tar.gz: 698eeeb6840e35554551cf2d87a2c2a567f8d5b58678a6d0eaed517781dff7db
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: c6f7efdd9f7ab1a12a53f84cb3bd539f1d2003f61098dee77a48b48a23bbb9e40814293ff6622477e1397a4857242a19775f2ca6d375df3f026208ef301f4600
|
|
7
|
+
data.tar.gz: d0351b5626ffbc7b17d88b60682a8d4e3b30bfa12d390d47cff858ca0cb741879f8182d2bace733f5e31f8e3a43e653e9e1ddf43be6f73206c87171c57aaf293
|
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
## [Unreleased]
|
|
2
|
+
|
|
3
|
+
## [0.4.0] - 2026-07-13
|
|
4
|
+
|
|
5
|
+
- **Renamed the gem from `noop-backup` to `boring-backup`.** Breaking, with no shims:
|
|
6
|
+
- `NoopBackup` namespace is now `BoringBackup`
|
|
7
|
+
- executable `nbu` is now `bb`
|
|
8
|
+
- `require "noop_backup"` is now `require "boring_backup"`
|
|
9
|
+
- env vars are prefixed `BB_` instead of `NBU_` (`BB_PREFIX`, `BB_MIN_SIZE`,
|
|
10
|
+
`BB_IGNORE_TABLES`, `BB_SENTINEL_KEY`, `BB_SENTINEL_HOST`, `BB_S3_PART_SIZE`,
|
|
11
|
+
`BB_S3_THREAD_COUNT`, `BB_S3_STORAGE_CLASS`)
|
|
12
|
+
- Configurable storage classes for s3
|
|
13
|
+
|
|
14
|
+
## [0.3.0] - 2026-07-13
|
|
15
|
+
|
|
16
|
+
- Make s3 upload stream part size and thread count configurable
|
|
17
|
+
- Sentinel integration
|
|
18
|
+
- Option to ignore tables
|
|
19
|
+
|
|
20
|
+
## [0.2.0] - 2026-07-04
|
|
21
|
+
|
|
22
|
+
- Support multiple destination stores in a single backup run
|
|
23
|
+
- Delete partial uploads and fail the job safely when a dump fails
|
|
24
|
+
- Validate configuration before running, with clearer errors
|
|
25
|
+
- A lot of undocumented API and configuration changes
|
|
26
|
+
|
|
27
|
+
## [0.1.3] - 2026-07-02
|
|
28
|
+
|
|
29
|
+
- Pass `access_key_id` and `secret_access_key` in config block
|
|
30
|
+
|
|
31
|
+
## [0.1.2] - 2026-07-02
|
|
32
|
+
|
|
33
|
+
- Only `exit 1` when running the CLI, not the backup service
|
|
34
|
+
- Automatically load BackupJob in Rails if using ActiveJob
|
|
35
|
+
|
|
36
|
+
## [0.1.1] - 2026-07-01
|
|
37
|
+
|
|
38
|
+
- Better S3 backup naming scheme - <prefix>/<database>/<YYYY>/<MM>/<dd-HHMMSS>.dump
|
|
39
|
+
- Fix bundler autoloading in projects
|
|
40
|
+
|
|
41
|
+
## [0.1.0] - 2026-07-01
|
|
42
|
+
|
|
43
|
+
- Initial release
|
|
44
|
+
- Only supports PostrgreSQL and AWS S3
|
data/LICENSE.txt
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
The MIT License (MIT)
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Georgi Mitrev
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in
|
|
13
|
+
all copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
|
21
|
+
THE SOFTWARE.
|
data/README.md
ADDED
|
@@ -0,0 +1,115 @@
|
|
|
1
|
+
# BoringBackup
|
|
2
|
+
|
|
3
|
+
Ruby gem for backing up PostgreSQL databases with minimum effort.
|
|
4
|
+
|
|
5
|
+
This gem is very much __unstable__. Expect APIs to change a lot before v1.0.0 is released.
|
|
6
|
+
|
|
7
|
+
## Installation
|
|
8
|
+
|
|
9
|
+
If using bundler, add the gem to your `Gemfile`:
|
|
10
|
+
|
|
11
|
+
```bash
|
|
12
|
+
bundle add boring-backup
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
If bundler is not being used to manage dependencies, install the gem by executing:
|
|
16
|
+
|
|
17
|
+
```bash
|
|
18
|
+
gem install boring-backup
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
The gem requires `pg_dump` to be installed on the machine that is running it.
|
|
22
|
+
|
|
23
|
+
## Usage
|
|
24
|
+
|
|
25
|
+
### Automatic
|
|
26
|
+
|
|
27
|
+
#### Rails 8 with Solid Queue
|
|
28
|
+
|
|
29
|
+
Add the BoringBackup job to the `config/recurring.yml` file:
|
|
30
|
+
|
|
31
|
+
```yaml
|
|
32
|
+
production:
|
|
33
|
+
# other jobs
|
|
34
|
+
boring_backup:
|
|
35
|
+
class: BoringBackup::BackupJob
|
|
36
|
+
schedule: at 3am every day
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
The job will run on the `:default` queue on your desired schedule.
|
|
40
|
+
If you wish to customize the job, create a new one and make sure to invoke
|
|
41
|
+
`BoringBackup::Commands::Backup.execute` in the `perform` method.
|
|
42
|
+
|
|
43
|
+
|
|
44
|
+
#### Sidekiq, Good Job, whenever
|
|
45
|
+
|
|
46
|
+
Create a new recurring job and have it invoke `BoringBackup::Commands::Backup.execute`
|
|
47
|
+
|
|
48
|
+
### Manually
|
|
49
|
+
|
|
50
|
+
```sh
|
|
51
|
+
bundle exec bb backup
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
This command will dump the database and stream it to your configured destinations without writing
|
|
55
|
+
anything to disk. Use any scheduler or even cron to run it periodically.
|
|
56
|
+
|
|
57
|
+
## Configuration
|
|
58
|
+
|
|
59
|
+
All configuration options can be edited in the initializer:
|
|
60
|
+
|
|
61
|
+
```rb
|
|
62
|
+
# config/initializers/boring-backup.rb
|
|
63
|
+
|
|
64
|
+
BoringBackup.configure do |config|
|
|
65
|
+
config.register(:s3) do |store|
|
|
66
|
+
store.bucket = Settings.aws.bucket
|
|
67
|
+
store.region = Settings.aws.region
|
|
68
|
+
store.access_key_id = Settings.aws.access_key_id
|
|
69
|
+
store.secret_access_key = Settings.aws.secret_access_key
|
|
70
|
+
end
|
|
71
|
+
|
|
72
|
+
config.notifier :slack do |slack|
|
|
73
|
+
slack.webhook_url = 'https://hooks.slack.com/services/whatever'
|
|
74
|
+
end
|
|
75
|
+
end
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
### Ignoring tables
|
|
79
|
+
|
|
80
|
+
To skip the rows of tables you don't need backed up, such as audit trails or job history:
|
|
81
|
+
|
|
82
|
+
```rb
|
|
83
|
+
BoringBackup.configure do |config|
|
|
84
|
+
config.ignore_tables = %w[versions logs]
|
|
85
|
+
end
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
Or set `BB_IGNORE_TABLES=versions,logs`.
|
|
89
|
+
|
|
90
|
+
A restore still creates these tables empty.
|
|
91
|
+
|
|
92
|
+
## Development
|
|
93
|
+
|
|
94
|
+
After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
|
|
95
|
+
|
|
96
|
+
To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
|
|
97
|
+
|
|
98
|
+
## Contributing
|
|
99
|
+
|
|
100
|
+
Bug reports and pull requests are welcome on GitHub at https://github.com/gmitrev/boring-backup.
|
|
101
|
+
|
|
102
|
+
## Wishlist
|
|
103
|
+
|
|
104
|
+
- [ ] S3-compatible backends
|
|
105
|
+
- [ ] file backend
|
|
106
|
+
- [ ] email notifier
|
|
107
|
+
- [ ] SMS notifier
|
|
108
|
+
- [ ] restore command
|
|
109
|
+
- [ ] test command
|
|
110
|
+
- [ ] encryption
|
|
111
|
+
- [ ] delete stale
|
|
112
|
+
|
|
113
|
+
## License
|
|
114
|
+
|
|
115
|
+
The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
|
data/Rakefile
ADDED
data/exe/bb
ADDED
|
@@ -0,0 +1 @@
|
|
|
1
|
+
require_relative "boring_backup"
|
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
require "thor"
|
|
2
|
+
|
|
3
|
+
module BoringBackup
|
|
4
|
+
class CLI < Thor
|
|
5
|
+
def self.exit_on_failure? = true
|
|
6
|
+
|
|
7
|
+
desc "backup", "Create and store a new backup"
|
|
8
|
+
def backup
|
|
9
|
+
BoringBackup.prepare!
|
|
10
|
+
result = BoringBackup::Commands::Backup.execute
|
|
11
|
+
exit 1 unless result.success?
|
|
12
|
+
rescue => e
|
|
13
|
+
warn e
|
|
14
|
+
exit 1
|
|
15
|
+
end
|
|
16
|
+
end
|
|
17
|
+
end
|
|
@@ -0,0 +1,135 @@
|
|
|
1
|
+
require "open3"
|
|
2
|
+
|
|
3
|
+
module BoringBackup::Commands
|
|
4
|
+
CommandResult = Struct.new(:error, :store_results, keyword_init: true) do
|
|
5
|
+
def success?
|
|
6
|
+
status == :success
|
|
7
|
+
end
|
|
8
|
+
|
|
9
|
+
def status
|
|
10
|
+
return :error if error
|
|
11
|
+
return :error if store_results.empty? || store_results.none?(&:success)
|
|
12
|
+
return :success if store_results.all?(&:success)
|
|
13
|
+
|
|
14
|
+
:partial_success
|
|
15
|
+
end
|
|
16
|
+
|
|
17
|
+
def messages
|
|
18
|
+
return ["❌ Fatal error: #{error.message}"] if error
|
|
19
|
+
|
|
20
|
+
store_results.map(&:message)
|
|
21
|
+
end
|
|
22
|
+
|
|
23
|
+
def report
|
|
24
|
+
BoringBackup.notify(self)
|
|
25
|
+
end
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
class Backup
|
|
29
|
+
def self.execute(report: BoringBackup.config.report?)
|
|
30
|
+
result = new.execute
|
|
31
|
+
|
|
32
|
+
result.report if report
|
|
33
|
+
|
|
34
|
+
result
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
def initialize
|
|
38
|
+
@key = generate_key
|
|
39
|
+
@store_results = []
|
|
40
|
+
@sinks = []
|
|
41
|
+
end
|
|
42
|
+
|
|
43
|
+
def execute
|
|
44
|
+
perform_sanity_check!
|
|
45
|
+
|
|
46
|
+
commands = [config.dump_command]
|
|
47
|
+
|
|
48
|
+
# Pipe pg_dump through pv if installed for a basic progress report
|
|
49
|
+
commands << ["pv", "-btra"] if config.report? && system("which", "pv", out: File::NULL, err: File::NULL)
|
|
50
|
+
|
|
51
|
+
Open3.pipeline_r(*commands) do |last_stdout, wait_threads|
|
|
52
|
+
@sinks = config.stores.map do |store|
|
|
53
|
+
reader, writer = IO.pipe(binmode: true)
|
|
54
|
+
|
|
55
|
+
thread = Thread.new do
|
|
56
|
+
store.backup!(@key, reader)
|
|
57
|
+
rescue => e
|
|
58
|
+
# *always* return a Result, even if unexpected. Add store name for debugging.
|
|
59
|
+
BoringBackup::Stores::Result.new(success: false, error: e, store: store.class.name, key: @key)
|
|
60
|
+
ensure
|
|
61
|
+
reader.close
|
|
62
|
+
end
|
|
63
|
+
|
|
64
|
+
BoringBackup::Tee::Sink.new(store:, writer:, thread:)
|
|
65
|
+
end
|
|
66
|
+
|
|
67
|
+
sinks_fanout = BoringBackup::Tee.new(@sinks)
|
|
68
|
+
|
|
69
|
+
begin
|
|
70
|
+
IO.copy_stream(last_stdout, sinks_fanout)
|
|
71
|
+
rescue => e
|
|
72
|
+
raise BoringBackup::DumpFailedError, "streaming failed: #{e.message}"
|
|
73
|
+
ensure
|
|
74
|
+
@sinks.each(&:close)
|
|
75
|
+
end
|
|
76
|
+
|
|
77
|
+
@store_results = @sinks.map(&:collect)
|
|
78
|
+
|
|
79
|
+
raise BoringBackup::DumpFailedError, "pipeline failed" unless wait_threads.all? { |t| t.value.success? }
|
|
80
|
+
end
|
|
81
|
+
|
|
82
|
+
CommandResult.new(store_results: @store_results)
|
|
83
|
+
rescue BoringBackup::DumpFailedError => error
|
|
84
|
+
# The dump stream itself failed, so uploads that finished are truncated copies of a bad stream — delete them.
|
|
85
|
+
# Collect the results first: if copy_stream raised, the collection line above never ran. Sinks are already
|
|
86
|
+
# closed on every DumpFailedError path, so collect can't block.
|
|
87
|
+
@store_results = @sinks.map(&:collect)
|
|
88
|
+
|
|
89
|
+
cleanup_uploaded_stores!
|
|
90
|
+
|
|
91
|
+
CommandResult.new(error:, store_results: @store_results)
|
|
92
|
+
rescue => error
|
|
93
|
+
CommandResult.new(error:, store_results: @store_results)
|
|
94
|
+
end
|
|
95
|
+
|
|
96
|
+
private
|
|
97
|
+
|
|
98
|
+
# 1. Check if any stores are registered.
|
|
99
|
+
# 2. Make sure **all** stores have a valid configuration
|
|
100
|
+
def perform_sanity_check!
|
|
101
|
+
raise BoringBackup::ConfigurationError, "No backup stores registered" if config.stores.empty?
|
|
102
|
+
raise BoringBackup::ConfigurationError, "Could not resolve PGDATABASE" if config.pg_env["PGDATABASE"].to_s.empty?
|
|
103
|
+
|
|
104
|
+
config.stores.each(&:validate!)
|
|
105
|
+
end
|
|
106
|
+
|
|
107
|
+
def cleanup_uploaded_stores!
|
|
108
|
+
@sinks.each do |sink|
|
|
109
|
+
sink.store.cleanup!(@key) if sink.collect&.success
|
|
110
|
+
rescue => e
|
|
111
|
+
warn "⚠️ Cleanup failed for #{sink.store.class}: #{e.message}"
|
|
112
|
+
end
|
|
113
|
+
end
|
|
114
|
+
|
|
115
|
+
def config
|
|
116
|
+
@config ||= BoringBackup.config
|
|
117
|
+
end
|
|
118
|
+
|
|
119
|
+
# File name of the current backup. Example:
|
|
120
|
+
#
|
|
121
|
+
# prefix db_name y m d h m s ms
|
|
122
|
+
# /database/db_name_production/2026/07/03-14-47-23-724.dump
|
|
123
|
+
def generate_key
|
|
124
|
+
now = Time.now.utc
|
|
125
|
+
|
|
126
|
+
[
|
|
127
|
+
config.prefix,
|
|
128
|
+
config.pg_env["PGDATABASE"],
|
|
129
|
+
now.strftime("%Y"),
|
|
130
|
+
now.strftime("%m"),
|
|
131
|
+
"#{now.strftime("%d-%H-%M-%S-%L")}.dump"
|
|
132
|
+
].compact.join("/")
|
|
133
|
+
end
|
|
134
|
+
end
|
|
135
|
+
end
|
|
@@ -0,0 +1,105 @@
|
|
|
1
|
+
module BoringBackup
|
|
2
|
+
class Configuration
|
|
3
|
+
DEFAULT_SENTINEL_HOST = "https://boringbackup.com"
|
|
4
|
+
|
|
5
|
+
attr_accessor :prefix,
|
|
6
|
+
:min_size,
|
|
7
|
+
:pg_host,
|
|
8
|
+
:pg_port,
|
|
9
|
+
:pg_user,
|
|
10
|
+
:pg_password,
|
|
11
|
+
:pg_database,
|
|
12
|
+
:sentinel_key
|
|
13
|
+
|
|
14
|
+
attr_reader :stores
|
|
15
|
+
attr_writer :report, :dump_command, :ignore_tables, :sentinel_host
|
|
16
|
+
|
|
17
|
+
def initialize
|
|
18
|
+
@prefix = ENV.fetch("BB_PREFIX", "database")
|
|
19
|
+
@stores = []
|
|
20
|
+
@min_size = ENV.fetch("BB_MIN_SIZE", 2048).to_i
|
|
21
|
+
@notifiers = [BoringBackup::Notifiers::Stdout.new]
|
|
22
|
+
@report = true
|
|
23
|
+
@ignore_tables = ENV.fetch("BB_IGNORE_TABLES", "").split(",")
|
|
24
|
+
@sentinel_key = ENV["BB_SENTINEL_KEY"]
|
|
25
|
+
@sentinel_host = ENV.fetch("BB_SENTINEL_HOST", DEFAULT_SENTINEL_HOST)
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
def report?
|
|
29
|
+
@report
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
def sentinel_host
|
|
33
|
+
@sentinel_host || DEFAULT_SENTINEL_HOST
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
def sentinel?
|
|
37
|
+
!sentinel_key.to_s.empty?
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
def sentinel
|
|
41
|
+
return unless sentinel?
|
|
42
|
+
|
|
43
|
+
@sentinel ||= BoringBackup::Notifiers::Sentinel.new(key: sentinel_key, host: sentinel_host)
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
def notifiers
|
|
47
|
+
[*@notifiers, sentinel].compact
|
|
48
|
+
end
|
|
49
|
+
|
|
50
|
+
def ignore_tables
|
|
51
|
+
Array(@ignore_tables).map { |table| table.to_s.strip }.reject(&:empty?).uniq
|
|
52
|
+
end
|
|
53
|
+
|
|
54
|
+
def dump_command
|
|
55
|
+
@dump_command || [
|
|
56
|
+
pg_env,
|
|
57
|
+
"pg_dump",
|
|
58
|
+
"--format=custom",
|
|
59
|
+
"--no-owner",
|
|
60
|
+
*ignore_tables.map { |table| "--exclude-table-data=#{table}" }
|
|
61
|
+
]
|
|
62
|
+
end
|
|
63
|
+
|
|
64
|
+
def register(store_type)
|
|
65
|
+
raise BoringBackup::ConfigurationError, "`config.register` requires a block" unless block_given?
|
|
66
|
+
|
|
67
|
+
store =
|
|
68
|
+
case store_type.to_sym
|
|
69
|
+
when :s3 then BoringBackup::Stores::S3.new
|
|
70
|
+
else raise BoringBackup::ConfigurationError, "unknown store type: #{store_type}"
|
|
71
|
+
end
|
|
72
|
+
|
|
73
|
+
@stores << store
|
|
74
|
+
|
|
75
|
+
yield store
|
|
76
|
+
end
|
|
77
|
+
|
|
78
|
+
def notifier(type)
|
|
79
|
+
case type
|
|
80
|
+
when :slack
|
|
81
|
+
notifier = BoringBackup::Notifiers::Slack.new
|
|
82
|
+
|
|
83
|
+
yield notifier
|
|
84
|
+
|
|
85
|
+
@notifiers << notifier
|
|
86
|
+
else raise "Unknown notifier: #{type}"
|
|
87
|
+
end
|
|
88
|
+
end
|
|
89
|
+
|
|
90
|
+
def pg_env
|
|
91
|
+
{
|
|
92
|
+
"PGHOST" => (pg_host || db_config[:host])&.to_s,
|
|
93
|
+
"PGPORT" => (pg_port || db_config[:port])&.to_s,
|
|
94
|
+
"PGUSER" => (pg_user || db_config[:username])&.to_s,
|
|
95
|
+
"PGPASSWORD" => (pg_password || db_config[:password])&.to_s,
|
|
96
|
+
"PGDATABASE" => (pg_database || db_config[:database])&.to_s
|
|
97
|
+
}.compact
|
|
98
|
+
end
|
|
99
|
+
|
|
100
|
+
def db_config
|
|
101
|
+
@db_config ||= defined?(::ActiveRecord) ?
|
|
102
|
+
::ActiveRecord::Base.connection_db_config.configuration_hash : {}
|
|
103
|
+
end
|
|
104
|
+
end
|
|
105
|
+
end
|
|
@@ -0,0 +1,103 @@
|
|
|
1
|
+
require "net/http"
|
|
2
|
+
require "uri"
|
|
3
|
+
require "json"
|
|
4
|
+
|
|
5
|
+
module BoringBackup::Notifiers
|
|
6
|
+
class Sentinel
|
|
7
|
+
PAYLOAD_VERSION = 1
|
|
8
|
+
|
|
9
|
+
OPEN_TIMEOUT = 5
|
|
10
|
+
READ_TIMEOUT = 10
|
|
11
|
+
MAX_ATTEMPTS = 3
|
|
12
|
+
|
|
13
|
+
LOCAL_HOSTS = %w[localhost 127.0.0.1 0.0.0.0]
|
|
14
|
+
|
|
15
|
+
attr_accessor :key, :host
|
|
16
|
+
|
|
17
|
+
def initialize(key:, host:)
|
|
18
|
+
@key = key
|
|
19
|
+
@host = host
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
def ping_url
|
|
23
|
+
URI.join(normalized_host, "/ping/#{key}")
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
def notify(result)
|
|
27
|
+
body = payload(result).to_json
|
|
28
|
+
attempt = 0
|
|
29
|
+
|
|
30
|
+
begin
|
|
31
|
+
attempt += 1
|
|
32
|
+
response = post(body)
|
|
33
|
+
|
|
34
|
+
return true if response.is_a?(Net::HTTPSuccess)
|
|
35
|
+
raise BoringBackup::Error, "#{response.code} #{response.body}" if retryable?(response)
|
|
36
|
+
|
|
37
|
+
warn "Sentinel ping failed: #{response.code} #{response.body}"
|
|
38
|
+
|
|
39
|
+
false
|
|
40
|
+
rescue => e
|
|
41
|
+
if attempt < MAX_ATTEMPTS
|
|
42
|
+
sleep(2**(attempt - 1))
|
|
43
|
+
retry
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
warn "Sentinel ping error after #{attempt} attempts: #{e.message}"
|
|
47
|
+
|
|
48
|
+
false
|
|
49
|
+
end
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
private
|
|
53
|
+
|
|
54
|
+
def normalized_host
|
|
55
|
+
uri = URI.parse(host)
|
|
56
|
+
|
|
57
|
+
return host if uri.is_a?(URI::HTTP)
|
|
58
|
+
|
|
59
|
+
"#{LOCAL_HOSTS.include?(uri.scheme) ? "http" : "https"}://#{host}"
|
|
60
|
+
end
|
|
61
|
+
|
|
62
|
+
def post(body)
|
|
63
|
+
uri = ping_url
|
|
64
|
+
|
|
65
|
+
http = Net::HTTP.new(uri.host, uri.port)
|
|
66
|
+
http.use_ssl = uri.scheme == "https"
|
|
67
|
+
http.open_timeout = OPEN_TIMEOUT
|
|
68
|
+
http.read_timeout = READ_TIMEOUT
|
|
69
|
+
|
|
70
|
+
request = Net::HTTP::Post.new(uri.request_uri, "Content-Type" => "application/json")
|
|
71
|
+
request.body = body
|
|
72
|
+
|
|
73
|
+
http.request(request)
|
|
74
|
+
end
|
|
75
|
+
|
|
76
|
+
def retryable?(response)
|
|
77
|
+
response.is_a?(Net::HTTPServerError) || response.is_a?(Net::HTTPTooManyRequests)
|
|
78
|
+
end
|
|
79
|
+
|
|
80
|
+
def payload(result)
|
|
81
|
+
store_results = result.store_results
|
|
82
|
+
|
|
83
|
+
{
|
|
84
|
+
version: PAYLOAD_VERSION,
|
|
85
|
+
status: result.status,
|
|
86
|
+
error: result.error&.message,
|
|
87
|
+
database: BoringBackup.config.pg_env["PGDATABASE"],
|
|
88
|
+
bytes: store_results.filter_map(&:bytes).max,
|
|
89
|
+
duration: store_results.filter_map(&:duration).max&.round(3),
|
|
90
|
+
stores: store_results.map do |store_result|
|
|
91
|
+
{
|
|
92
|
+
store: store_result.store,
|
|
93
|
+
success: store_result.success,
|
|
94
|
+
key: store_result.key,
|
|
95
|
+
bytes: store_result.bytes,
|
|
96
|
+
duration: store_result.duration&.round(3),
|
|
97
|
+
error: store_result.error&.message
|
|
98
|
+
}
|
|
99
|
+
end
|
|
100
|
+
}
|
|
101
|
+
end
|
|
102
|
+
end
|
|
103
|
+
end
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
require "net/http"
|
|
2
|
+
require "uri"
|
|
3
|
+
require "json"
|
|
4
|
+
|
|
5
|
+
module BoringBackup::Notifiers
|
|
6
|
+
class Slack
|
|
7
|
+
OPEN_TIMEOUT = 5
|
|
8
|
+
READ_TIMEOUT = 10
|
|
9
|
+
|
|
10
|
+
attr_accessor :webhook_url
|
|
11
|
+
|
|
12
|
+
def notify(result)
|
|
13
|
+
uri = URI(webhook_url)
|
|
14
|
+
http = Net::HTTP.new(uri.host, uri.port)
|
|
15
|
+
http.use_ssl = true
|
|
16
|
+
http.open_timeout = OPEN_TIMEOUT
|
|
17
|
+
http.read_timeout = READ_TIMEOUT
|
|
18
|
+
|
|
19
|
+
request = Net::HTTP::Post.new(uri.request_uri, "Content-Type" => "application/json")
|
|
20
|
+
request.body = {text: result.messages.join("\n")}.to_json
|
|
21
|
+
|
|
22
|
+
response = http.request(request)
|
|
23
|
+
|
|
24
|
+
success = response.is_a?(Net::HTTPSuccess)
|
|
25
|
+
|
|
26
|
+
warn "Slack notify failed: #{response.code} #{response.body}" unless success
|
|
27
|
+
|
|
28
|
+
success
|
|
29
|
+
rescue => e
|
|
30
|
+
# Never fail a backup because of a notification error
|
|
31
|
+
warn "Slack notify error: #{e.message}"
|
|
32
|
+
end
|
|
33
|
+
end
|
|
34
|
+
end
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
module BoringBackup::Stores
|
|
2
|
+
Result = Struct.new(:success, :error, :store, :key, :bytes, :duration, keyword_init: true) do
|
|
3
|
+
def message
|
|
4
|
+
if success
|
|
5
|
+
"✅ [#{store}] Backup successful — #{BoringBackup.utils.human_size(bytes)} in #{duration.round(1)}s → /#{key}"
|
|
6
|
+
else
|
|
7
|
+
"❌ [#{store}] Backup failed: #{error.message}"
|
|
8
|
+
end
|
|
9
|
+
end
|
|
10
|
+
end
|
|
11
|
+
end
|
|
@@ -0,0 +1,119 @@
|
|
|
1
|
+
require "aws-sdk-s3"
|
|
2
|
+
|
|
3
|
+
module BoringBackup::Stores
|
|
4
|
+
class S3 < Store
|
|
5
|
+
DEFAULT_STORAGE_CLASS = :standard_ia
|
|
6
|
+
|
|
7
|
+
STORAGE_CLASSES = %i[
|
|
8
|
+
standard
|
|
9
|
+
standard_ia
|
|
10
|
+
onezone_ia
|
|
11
|
+
intelligent_tiering
|
|
12
|
+
glacier_ir
|
|
13
|
+
glacier
|
|
14
|
+
deep_archive
|
|
15
|
+
reduced_redundancy
|
|
16
|
+
express_onezone
|
|
17
|
+
outposts
|
|
18
|
+
snow
|
|
19
|
+
].freeze
|
|
20
|
+
|
|
21
|
+
attr_accessor :bucket, :region, :access_key_id, :secret_access_key, :part_size, :thread_count
|
|
22
|
+
attr_writer :storage_class
|
|
23
|
+
|
|
24
|
+
def initialize
|
|
25
|
+
@bucket = ENV["AWS_S3_BUCKET"]
|
|
26
|
+
@region = ENV["AWS_REGION"]
|
|
27
|
+
@access_key_id = ENV["AWS_ACCESS_KEY_ID"]
|
|
28
|
+
@secret_access_key = ENV["AWS_SECRET_ACCESS_KEY"]
|
|
29
|
+
@part_size = ENV.fetch("BB_S3_PART_SIZE", 8 * 1024 * 1024).to_i
|
|
30
|
+
@thread_count = ENV.fetch("BB_S3_THREAD_COUNT", 2).to_i
|
|
31
|
+
@storage_class = ENV.fetch("BB_S3_STORAGE_CLASS", DEFAULT_STORAGE_CLASS)
|
|
32
|
+
end
|
|
33
|
+
|
|
34
|
+
def storage_class
|
|
35
|
+
value = @storage_class.to_s.strip.downcase
|
|
36
|
+
|
|
37
|
+
value.empty? ? nil : value.to_sym
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
# Prefer Aws::S3::TransferManager for streaming uploads if available.
|
|
41
|
+
# Aws::S3::Resource.upload_stream is deprecated in newer versions
|
|
42
|
+
def backup!(key, stream)
|
|
43
|
+
started = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
44
|
+
bytes = 0
|
|
45
|
+
|
|
46
|
+
validate!
|
|
47
|
+
|
|
48
|
+
upload_attempted = true
|
|
49
|
+
|
|
50
|
+
if defined?(Aws::S3::TransferManager)
|
|
51
|
+
manager = Aws::S3::TransferManager.new(client: s3_client)
|
|
52
|
+
|
|
53
|
+
manager.upload_stream(bucket:, key: key, **upload_options) do |s3_stream|
|
|
54
|
+
bytes = IO.copy_stream(stream, s3_stream)
|
|
55
|
+
end
|
|
56
|
+
else
|
|
57
|
+
object = Aws::S3::Resource.new(client: s3_client).bucket(bucket).object(key)
|
|
58
|
+
|
|
59
|
+
object.upload_stream(**upload_options) do |s3_stream|
|
|
60
|
+
bytes = IO.copy_stream(stream, s3_stream)
|
|
61
|
+
end
|
|
62
|
+
end
|
|
63
|
+
|
|
64
|
+
duration = Process.clock_gettime(Process::CLOCK_MONOTONIC) - started
|
|
65
|
+
|
|
66
|
+
if config.min_size && bytes < config.min_size.to_i
|
|
67
|
+
raise BoringBackup::DumpTooSmallError, "backup too small: #{BoringBackup.utils.human_size(bytes)} < min_size (#{BoringBackup.utils.human_size(config.min_size)})"
|
|
68
|
+
end
|
|
69
|
+
|
|
70
|
+
Result.new(success: true, store: :s3, bytes:, key:, duration:)
|
|
71
|
+
rescue => e
|
|
72
|
+
duration = Process.clock_gettime(Process::CLOCK_MONOTONIC) - started
|
|
73
|
+
|
|
74
|
+
cleanup!(key) if upload_attempted
|
|
75
|
+
|
|
76
|
+
Result.new(success: false, error: e, store: :s3, bytes:, key:, duration:)
|
|
77
|
+
end
|
|
78
|
+
|
|
79
|
+
def validate!
|
|
80
|
+
raise BoringBackup::ConfigurationError, "bucket is not configured" if bucket.to_s.empty?
|
|
81
|
+
|
|
82
|
+
if access_key_id.to_s.empty? != secret_access_key.to_s.empty?
|
|
83
|
+
raise BoringBackup::ConfigurationError,
|
|
84
|
+
"access_key_id and secret_access_key must both be set, or both left blank to use the default AWS credential chain"
|
|
85
|
+
end
|
|
86
|
+
|
|
87
|
+
if storage_class && !STORAGE_CLASSES.include?(storage_class)
|
|
88
|
+
raise BoringBackup::ConfigurationError,
|
|
89
|
+
"unknown storage class: #{storage_class.inspect} (expected one of: #{STORAGE_CLASSES.map(&:inspect).join(", ")}, or nil to use the bucket default)"
|
|
90
|
+
end
|
|
91
|
+
end
|
|
92
|
+
|
|
93
|
+
def cleanup!(key)
|
|
94
|
+
s3_client.delete_object(bucket:, key:)
|
|
95
|
+
rescue => e
|
|
96
|
+
warn "Failed to clean up partial upload #{key}: #{e.message}"
|
|
97
|
+
end
|
|
98
|
+
|
|
99
|
+
private
|
|
100
|
+
|
|
101
|
+
def upload_options
|
|
102
|
+
options = {part_size:, thread_count:}
|
|
103
|
+
options[:storage_class] = storage_class.to_s.upcase if storage_class
|
|
104
|
+
options
|
|
105
|
+
end
|
|
106
|
+
|
|
107
|
+
def s3_client
|
|
108
|
+
@_s3_client ||= Aws::S3::Client.new(**s3_config)
|
|
109
|
+
end
|
|
110
|
+
|
|
111
|
+
def s3_config
|
|
112
|
+
{
|
|
113
|
+
region: region,
|
|
114
|
+
access_key_id: access_key_id,
|
|
115
|
+
secret_access_key: secret_access_key
|
|
116
|
+
}.compact
|
|
117
|
+
end
|
|
118
|
+
end
|
|
119
|
+
end
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
module BoringBackup
|
|
2
|
+
class Tee
|
|
3
|
+
Sink = Struct.new(:store, :writer, :thread, keyword_init: true) do
|
|
4
|
+
def write(chunk)
|
|
5
|
+
return if @error
|
|
6
|
+
|
|
7
|
+
writer.write(chunk)
|
|
8
|
+
rescue Errno::EPIPE, IOError => e
|
|
9
|
+
@error = e
|
|
10
|
+
|
|
11
|
+
close unless writer.closed?
|
|
12
|
+
end
|
|
13
|
+
|
|
14
|
+
def close
|
|
15
|
+
writer.close
|
|
16
|
+
end
|
|
17
|
+
|
|
18
|
+
def collect
|
|
19
|
+
thread.value
|
|
20
|
+
rescue
|
|
21
|
+
nil
|
|
22
|
+
end
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
def initialize(sinks) = @sinks = sinks
|
|
26
|
+
|
|
27
|
+
def write(chunk)
|
|
28
|
+
@sinks.each { |sink| sink.write(chunk) }
|
|
29
|
+
|
|
30
|
+
chunk.bytesize
|
|
31
|
+
end
|
|
32
|
+
end
|
|
33
|
+
end
|
|
@@ -0,0 +1,68 @@
|
|
|
1
|
+
require_relative "boring_backup/version"
|
|
2
|
+
require_relative "boring_backup/configuration"
|
|
3
|
+
require_relative "boring_backup/tee"
|
|
4
|
+
require_relative "boring_backup/utils"
|
|
5
|
+
require_relative "boring_backup/stores"
|
|
6
|
+
require_relative "boring_backup/commands/backup"
|
|
7
|
+
require_relative "boring_backup/notifiers/slack"
|
|
8
|
+
require_relative "boring_backup/notifiers/stdout"
|
|
9
|
+
require_relative "boring_backup/notifiers/sentinel"
|
|
10
|
+
|
|
11
|
+
module BoringBackup
|
|
12
|
+
class Error < StandardError; end
|
|
13
|
+
|
|
14
|
+
class DumpTooSmallError < Error; end
|
|
15
|
+
class DumpFailedError < Error; end
|
|
16
|
+
class ConfigurationError < Error; end
|
|
17
|
+
|
|
18
|
+
class BackupFailedError < Error
|
|
19
|
+
attr_reader :result
|
|
20
|
+
|
|
21
|
+
def initialize(result)
|
|
22
|
+
@result = result
|
|
23
|
+
|
|
24
|
+
super("Backup failed (#{result.status}): #{result.error&.message || failed_stores(result)}")
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
private
|
|
28
|
+
|
|
29
|
+
def failed_stores(result)
|
|
30
|
+
result.store_results.reject(&:success).map(&:store).join(", ")
|
|
31
|
+
end
|
|
32
|
+
end
|
|
33
|
+
|
|
34
|
+
class << self
|
|
35
|
+
def configuration
|
|
36
|
+
@configuration ||= Configuration.new
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
alias_method :config, :configuration
|
|
40
|
+
|
|
41
|
+
def reset!
|
|
42
|
+
@configuration = nil
|
|
43
|
+
end
|
|
44
|
+
|
|
45
|
+
def configure
|
|
46
|
+
yield(configuration)
|
|
47
|
+
end
|
|
48
|
+
|
|
49
|
+
# Attempt to boot host app. Booting it should trigger the configuration process.
|
|
50
|
+
def prepare!
|
|
51
|
+
env_file = File.expand_path("config/environment.rb")
|
|
52
|
+
|
|
53
|
+
require env_file if File.exist?(env_file)
|
|
54
|
+
end
|
|
55
|
+
|
|
56
|
+
def notify(result)
|
|
57
|
+
config.notifiers.each do |notifier|
|
|
58
|
+
notifier.notify(result)
|
|
59
|
+
end
|
|
60
|
+
end
|
|
61
|
+
|
|
62
|
+
def utils
|
|
63
|
+
BoringBackup::Utils
|
|
64
|
+
end
|
|
65
|
+
end
|
|
66
|
+
end
|
|
67
|
+
|
|
68
|
+
require_relative "boring_backup/plugins/rails" if defined? Rails::Railtie
|
metadata
ADDED
|
@@ -0,0 +1,97 @@
|
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
|
2
|
+
name: boring-backup
|
|
3
|
+
version: !ruby/object:Gem::Version
|
|
4
|
+
version: 0.4.0
|
|
5
|
+
platform: ruby
|
|
6
|
+
authors:
|
|
7
|
+
- Georgi Mitrev
|
|
8
|
+
bindir: exe
|
|
9
|
+
cert_chain: []
|
|
10
|
+
date: 1980-01-02 00:00:00.000000000 Z
|
|
11
|
+
dependencies:
|
|
12
|
+
- !ruby/object:Gem::Dependency
|
|
13
|
+
name: thor
|
|
14
|
+
requirement: !ruby/object:Gem::Requirement
|
|
15
|
+
requirements:
|
|
16
|
+
- - ">="
|
|
17
|
+
- !ruby/object:Gem::Version
|
|
18
|
+
version: '0'
|
|
19
|
+
type: :runtime
|
|
20
|
+
prerelease: false
|
|
21
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
22
|
+
requirements:
|
|
23
|
+
- - ">="
|
|
24
|
+
- !ruby/object:Gem::Version
|
|
25
|
+
version: '0'
|
|
26
|
+
- !ruby/object:Gem::Dependency
|
|
27
|
+
name: aws-sdk-s3
|
|
28
|
+
requirement: !ruby/object:Gem::Requirement
|
|
29
|
+
requirements:
|
|
30
|
+
- - "~>"
|
|
31
|
+
- !ruby/object:Gem::Version
|
|
32
|
+
version: '1'
|
|
33
|
+
type: :runtime
|
|
34
|
+
prerelease: false
|
|
35
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
36
|
+
requirements:
|
|
37
|
+
- - "~>"
|
|
38
|
+
- !ruby/object:Gem::Version
|
|
39
|
+
version: '1'
|
|
40
|
+
description: The simplest way to add recurring database backups to your project with
|
|
41
|
+
minimal setup required.
|
|
42
|
+
email:
|
|
43
|
+
- gvmitrev@gmail.com
|
|
44
|
+
executables:
|
|
45
|
+
- bb
|
|
46
|
+
extensions: []
|
|
47
|
+
extra_rdoc_files: []
|
|
48
|
+
files:
|
|
49
|
+
- CHANGELOG.md
|
|
50
|
+
- LICENSE.txt
|
|
51
|
+
- README.md
|
|
52
|
+
- Rakefile
|
|
53
|
+
- exe/bb
|
|
54
|
+
- lib/boring-backup.rb
|
|
55
|
+
- lib/boring_backup.rb
|
|
56
|
+
- lib/boring_backup/cli.rb
|
|
57
|
+
- lib/boring_backup/commands/backup.rb
|
|
58
|
+
- lib/boring_backup/configuration.rb
|
|
59
|
+
- lib/boring_backup/jobs/backup_job.rb
|
|
60
|
+
- lib/boring_backup/notifiers/sentinel.rb
|
|
61
|
+
- lib/boring_backup/notifiers/slack.rb
|
|
62
|
+
- lib/boring_backup/notifiers/stdout.rb
|
|
63
|
+
- lib/boring_backup/plugins/rails.rb
|
|
64
|
+
- lib/boring_backup/stores.rb
|
|
65
|
+
- lib/boring_backup/stores/result.rb
|
|
66
|
+
- lib/boring_backup/stores/s3.rb
|
|
67
|
+
- lib/boring_backup/stores/store.rb
|
|
68
|
+
- lib/boring_backup/tee.rb
|
|
69
|
+
- lib/boring_backup/utils.rb
|
|
70
|
+
- lib/boring_backup/version.rb
|
|
71
|
+
homepage: https://github.com/gmitrev/boring-backup
|
|
72
|
+
licenses:
|
|
73
|
+
- MIT
|
|
74
|
+
metadata:
|
|
75
|
+
homepage_uri: https://github.com/gmitrev/boring-backup
|
|
76
|
+
source_code_uri: https://github.com/gmitrev/boring-backup/tree/main
|
|
77
|
+
changelog_uri: https://github.com/gmitrev/boring-backup/blob/main/CHANGELOG.md
|
|
78
|
+
rubygems_mfa_required: 'true'
|
|
79
|
+
allowed_push_host: https://rubygems.org
|
|
80
|
+
rdoc_options: []
|
|
81
|
+
require_paths:
|
|
82
|
+
- lib
|
|
83
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
|
84
|
+
requirements:
|
|
85
|
+
- - ">="
|
|
86
|
+
- !ruby/object:Gem::Version
|
|
87
|
+
version: 3.1.0
|
|
88
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
89
|
+
requirements:
|
|
90
|
+
- - ">="
|
|
91
|
+
- !ruby/object:Gem::Version
|
|
92
|
+
version: '0'
|
|
93
|
+
requirements: []
|
|
94
|
+
rubygems_version: 4.0.10
|
|
95
|
+
specification_version: 4
|
|
96
|
+
summary: Hassle-free database backups
|
|
97
|
+
test_files: []
|