bard-backup 0.16.3 → 0.17.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.ruby-version +1 -1
- data/CLAUDE.md +6 -3
- data/README.md +64 -1
- data/lib/bard/backup/cached_local_backhoe.rb +19 -8
- data/lib/bard/backup/deleter.rb +15 -8
- data/lib/bard/backup/destination/s3_destination.rb +2 -1
- data/lib/bard/backup/file_tree.rb +2 -2
- data/lib/bard/backup/local_backhoe.rb +7 -4
- data/lib/bard/backup/restore_check/scratch_record.rb +11 -0
- data/lib/bard/backup/restore_check.rb +141 -0
- data/lib/bard/backup/s3_tree.rb +11 -6
- data/lib/bard/backup/tasks.rake +8 -0
- data/lib/bard/backup/upload_coverage.rb +53 -0
- data/lib/bard/backup/version.rb +1 -1
- data/lib/bard/backup.rb +6 -0
- metadata +8 -5
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 44a2ad21a4947347c0bfee31ab392adc09d20a338af463a331cd66d834c3a76a
|
|
4
|
+
data.tar.gz: 52fc265e55825f3753c66bb30c411a3367b5e928bc8b639036e4e40f1cacdc48
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: dc47138a8c9f787e11d9aa7cac6748e27f25a7ca5bf6334dad91f2150fc00cd598a1ddc7f9e67cffbdf9bde8346b6f3143886e315f8ab30fa2d6c7aa360c83c1
|
|
7
|
+
data.tar.gz: 5b6a9ad1c165625f826ee05dda3d1b819e2dc6ac6df546445e46b10df082402b50e347788607f149a673a5d4e54e56737badb86470ce2698aa99956e89674dce
|
data/.ruby-version
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
ruby-
|
|
1
|
+
ruby-4.0.10
|
data/CLAUDE.md
CHANGED
|
@@ -7,6 +7,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
|
|
|
7
7
|
bard-backup is a Ruby gem that provides automated backup for Bard projects. It:
|
|
8
8
|
- Dumps the database, uploads to S3, deletes old backups using a retention heuristic (48 hours, 30 days, 26 weeks, 24 months, then yearly), and verifies the previous hour's backup exists.
|
|
9
9
|
- Syncs configured data directories to S3 via a manifest-cached file tree (`Bard::Backup::FileTree`).
|
|
10
|
+
- Checks on a schedule that the latest dump restores into a scratch database (`Bard::Backup::RestoreCheck`).
|
|
10
11
|
- Optionally encrypts uploaded payloads at rest with AES-256-GCM (`Bard::Backup::Encryptor`).
|
|
11
12
|
|
|
12
13
|
## Commands
|
|
@@ -61,14 +62,16 @@ Encryption at rest is opt-in per destination via an `encryption_key` in the `bar
|
|
|
61
62
|
credentials entry — there is no DSL toggle.
|
|
62
63
|
|
|
63
64
|
**Key classes**:
|
|
64
|
-
- `S3Tree` — `Data.define`-based S3 wrapper used by both `S3Destination` and `FileTree`. Methods: `list_objects`, `put_file`, `put_body`, `get`, `delete_keys`, `mv`, `empty
|
|
65
|
+
- `S3Tree` — `Data.define`-based S3 wrapper used by both `S3Destination` and `FileTree`. Methods: `list_objects`, `put_file`, `put_body`, `get`, `delete_keys`, `mv`, `empty!`, `object_lock_enabled?`. Supports encryption via `Encryptor`, STS `session_token`, and a `retain_until` that uploads under a `GOVERNANCE` Object Lock.
|
|
65
66
|
- `FileTree` — syncs local data paths to S3 using a local `.bard-file-tree-sync.json` manifest (mtime+size fast path, MD5 verification, falls back to remote listing on first run)
|
|
66
67
|
- `Encryptor` — AES-256-GCM with HKDF-derived keys and a deterministic IV (HMAC of plaintext), enabling content-addressable encryption
|
|
67
|
-
- `Deleter` — implements the retention policy via `Filter` structs that check time-based granularities
|
|
68
|
+
- `Deleter` — implements the retention policy via `Filter` structs that check time-based granularities. `Deleter.retain_until(timestamp)` gives the moment it first deletes a dump, which `S3Destination` uses as the Object Lock date when the bucket has Object Lock enabled
|
|
68
69
|
- `LocalBackhoe` / `CachedLocalBackhoe` — database dump strategies (cached variant avoids conflicts when running parallel destinations)
|
|
69
70
|
- `Finder` — lists backups across destinations and selects one: by timestamp (nearest match), the latest, or the most-recent as a `Bard::Backup` (with size + destination info). `#available` returns one `Bard::Backup` per timestamp (with size) for listing. Backs `Bard::Backup.available`/`.latest` and `Restore`
|
|
70
71
|
- `Restore` — downloads the backup selected by `Finder`, decrypts it via `S3Tree#get`, and loads it into the local database via backhoe (drop-and-create). Backs `Bard::Backup.restore!`
|
|
72
|
+
- `RestoreCheck` — restores the latest backup into a scratch database beside the live one (via `Backhoe::Load`), rejects corrupt or truncated dumps first, checks the restored tables hold data, drops the scratch database, and raises `RestoreCheck::Failed` listing every problem. Backs `Bard::Backup.check_restore!` and `bard:backup:check`
|
|
73
|
+
- `UploadCoverage` — raises `UncoveredUploads` when Active Storage's service keeps files on local disk outside the declared `data` paths; `bard:backup` and `bard:backup:data` run it after backing up
|
|
71
74
|
- `Rekey` — rewrites every object under an S3 path so it is encrypted under a key (from plaintext, or from an old key via `from_key:`), skipping objects already under it and reading each rewrite back. Backs `Bard::Backup.encrypt_existing!` and `bard:backup:encrypt`
|
|
72
75
|
- `BackupConfig` — the `backup do ... end` DSL surface (`bard`, `disabled`, `s3 name, **kwargs`); `create!` reads `bard_config.backup.destinations` from it
|
|
73
|
-
- `Railtie` — loads `tasks.rake` which provides `bard:backup` (DB + data), `bard:backup:data` (data only), `bard:backup:encrypt` (encrypt pre-key backups and data files), and `bard:backup:restore[at]` (restore/list) rake tasks in Rails apps
|
|
76
|
+
- `Railtie` — loads `tasks.rake` which provides `bard:backup` (DB + data), `bard:backup:data` (data only), `bard:backup:encrypt` (encrypt pre-key backups and data files), `bard:backup:check` (scheduled restore check), and `bard:backup:restore[at]` (restore/list) rake tasks in Rails apps
|
|
74
77
|
- `bard/plugins/backup_restore.rb` — registers the `bard backup restore [TIMESTAMP]` CLI subcommand (loaded by the `bard` CLI via `Gem.find_files`); shells out to the `bard:backup:restore` rake task
|
data/README.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
Bard::Backup handles backups for a bard project:
|
|
4
4
|
1. Takes a database dump and uploads it to S3 (or PUTs it to presigned URLs)
|
|
5
|
-
2. Syncs configured data directories to S3 with a local manifest cache, skipping the
|
|
5
|
+
2. Syncs configured data directories to S3 with a local manifest cache, skipping the SQLite databases Rails keeps in `storage/` (the database dump covers the app's own)
|
|
6
6
|
3. Deletes old database backups using a backoff heuristic: 48 hours, 30 days, 26 weeks, 24 months, then yearly
|
|
7
7
|
4. Raises an error if we don't have a database backup from the previous hour
|
|
8
8
|
5. Optionally encrypts uploaded payloads at rest with AES-256-GCM
|
|
@@ -50,8 +50,71 @@ Then run via the rake tasks provided by the bundled Railtie:
|
|
|
50
50
|
rake bard:backup # database backup + data file-tree sync
|
|
51
51
|
rake bard:backup:data # data file-tree sync only
|
|
52
52
|
rake bard:backup:encrypt # encrypt everything stored before encryption_key was added
|
|
53
|
+
rake bard:backup:check # prove the latest backup restores (see below)
|
|
53
54
|
```
|
|
54
55
|
|
|
56
|
+
Data directories are declared once in `config/bard.rb`, and `bard data` copies the same ones:
|
|
57
|
+
|
|
58
|
+
```ruby
|
|
59
|
+
data "public/userfiles", "storage"
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
Each sync uploads only new and changed files to `bard-data/<project>/` and deletes what was
|
|
63
|
+
removed locally (the bucket's versioning keeps the old copies). When Active Storage keeps
|
|
64
|
+
uploads on local disk outside every declared data path, `bard:backup` and `bard:backup:data`
|
|
65
|
+
raise `Bard::Backup::UncoveredUploads` after backing up everything else, so the gap shows up
|
|
66
|
+
in exception notifications instead of in a failed restore.
|
|
67
|
+
|
|
68
|
+
## Checking restores
|
|
69
|
+
|
|
70
|
+
`rake bard:backup:check` downloads the latest database backup, decrypts it, and loads it with
|
|
71
|
+
backhoe into a scratch database beside the live one (`<database>_restore_check`, or a temp
|
|
72
|
+
file for SQLite). It then checks that the restored database has tables, a non-empty
|
|
73
|
+
`schema_migrations`, and at least one table with rows. It drops the scratch database whatever
|
|
74
|
+
the outcome. Before loading, it rejects dumps that are not valid gzip, fail their CRC, have
|
|
75
|
+
bytes appended after the gzip stream, or end before mysqldump's or pg_dump's completion line.
|
|
76
|
+
Any problem raises `Bard::Backup::RestoreCheck::Failed`, naming the backup and every problem
|
|
77
|
+
found. On success it prints a line such as:
|
|
78
|
+
|
|
79
|
+
```
|
|
80
|
+
Restore check passed for backup 2026-11-18T03:00:00Z: 42 tables, 183920 rows
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
Schedule it daily, away from the hourly backup, with whenever:
|
|
84
|
+
|
|
85
|
+
```ruby
|
|
86
|
+
every 1.day, at: "3:30 am" do
|
|
87
|
+
rake "bard:backup:check"
|
|
88
|
+
end
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
or with Solid Queue in `config/recurring.yml`:
|
|
92
|
+
|
|
93
|
+
```yaml
|
|
94
|
+
production:
|
|
95
|
+
bard_backup_check:
|
|
96
|
+
command: system "bin/rake bard:backup:check"
|
|
97
|
+
schedule: every day at 3:30am
|
|
98
|
+
queue: serial
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
The database user needs permission to create and drop the scratch database, and the server
|
|
102
|
+
needs disk space for a second copy of the database while the check runs.
|
|
103
|
+
|
|
104
|
+
## Object Lock
|
|
105
|
+
|
|
106
|
+
When the destination bucket has S3 Object Lock enabled, each dump is uploaded under a
|
|
107
|
+
`GOVERNANCE` lock that lasts exactly as long as the retention policy keeps it: 48 hours for an
|
|
108
|
+
hourly dump, 30 days for the day's 00:00 dump, 26 weeks for Monday's, 24 months for the
|
|
109
|
+
1st of the month's, and 25 years for January 1st's. The hourly run still deletes dumps as they
|
|
110
|
+
age out, but in a locked bucket that only adds a delete marker, so a compromised server can
|
|
111
|
+
hide its backups but cannot destroy them before their lock ends. Pair it with a noncurrent
|
|
112
|
+
version lifecycle rule to clear each version once its lock lapses.
|
|
113
|
+
|
|
114
|
+
The gem detects the lock with `GetObjectLockConfiguration` on every run, so the backup's AWS
|
|
115
|
+
user needs `s3:GetBucketObjectLockConfiguration` on the bucket and `s3:PutObjectRetention` on
|
|
116
|
+
its prefix. Without the first, the backup fails rather than uploading an unlocked dump.
|
|
117
|
+
|
|
55
118
|
## Restoring
|
|
56
119
|
|
|
57
120
|
Restore a database backup from S3 into the local database. The backup nearest the
|
|
@@ -1,26 +1,37 @@
|
|
|
1
1
|
require "backhoe"
|
|
2
|
+
require "bard/config"
|
|
3
|
+
require "fileutils"
|
|
4
|
+
require "tmpdir"
|
|
2
5
|
|
|
3
6
|
module Bard
|
|
4
7
|
class Backup
|
|
5
8
|
class CachedLocalBackhoe < Struct.new(:s3_tree, :now)
|
|
9
|
+
class << self
|
|
10
|
+
attr_accessor :dump_dir
|
|
11
|
+
end
|
|
12
|
+
|
|
6
13
|
def self.call *args
|
|
7
14
|
new(*args).call
|
|
8
15
|
end
|
|
9
16
|
|
|
10
17
|
def call
|
|
11
|
-
s3_tree.put_file(path,
|
|
18
|
+
s3_tree.put_file(path, filename)
|
|
12
19
|
end
|
|
13
20
|
|
|
14
21
|
private
|
|
15
22
|
|
|
23
|
+
def filename
|
|
24
|
+
"#{now.iso8601}.sql.gz"
|
|
25
|
+
end
|
|
26
|
+
|
|
16
27
|
def path
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
28
|
+
cached = self.class.dump_dir && File.join(self.class.dump_dir, filename)
|
|
29
|
+
return cached if cached && File.exist?(cached)
|
|
30
|
+
|
|
31
|
+
FileUtils.rm_rf(self.class.dump_dir) if self.class.dump_dir
|
|
32
|
+
self.class.dump_dir = dir = Dir.mktmpdir("#{Bard::Config.current.project_name}-backup-")
|
|
33
|
+
at_exit { FileUtils.rm_rf dir }
|
|
34
|
+
File.join(dir, filename).tap { |path| Backhoe.dump path }
|
|
24
35
|
end
|
|
25
36
|
end
|
|
26
37
|
end
|
data/lib/bard/backup/deleter.rb
CHANGED
|
@@ -5,19 +5,22 @@ require "active_support/core_ext/integer/time"
|
|
|
5
5
|
module Bard
|
|
6
6
|
class Backup
|
|
7
7
|
class Deleter < Struct.new(:s3_tree, :now)
|
|
8
|
+
WINDOWS = [[48, :hours], [30, :days], [26, :weeks], [24, :months], [25, :years]].freeze
|
|
9
|
+
private_constant :WINDOWS
|
|
10
|
+
|
|
11
|
+
# The moment the hourly run first deletes a dump taken at +timestamp+.
|
|
12
|
+
def self.retain_until(timestamp)
|
|
13
|
+
WINDOWS.filter_map { |limit, unit| Filter.new(nil, limit, unit).retain_until(timestamp) }.max
|
|
14
|
+
end
|
|
15
|
+
|
|
8
16
|
def call
|
|
9
17
|
s3_tree.delete_keys files_to_delete
|
|
10
18
|
end
|
|
11
19
|
|
|
12
20
|
def files_to_delete
|
|
21
|
+
filters = WINDOWS.map { |limit, unit| Filter.new(now, limit, unit) }
|
|
13
22
|
s3_tree.list_objects.keys.select do |file|
|
|
14
|
-
|
|
15
|
-
Filter.new(now, 48, :hours),
|
|
16
|
-
Filter.new(now, 30, :days),
|
|
17
|
-
Filter.new(now, 26, :weeks),
|
|
18
|
-
Filter.new(now, 24, :months),
|
|
19
|
-
Filter.new(now, 25, :years),
|
|
20
|
-
].all? { |filter| !filter.cover?(file) }
|
|
23
|
+
filters.all? { |filter| !filter.cover?(file) }
|
|
21
24
|
end
|
|
22
25
|
end
|
|
23
26
|
|
|
@@ -29,6 +32,11 @@ module Bard
|
|
|
29
32
|
end
|
|
30
33
|
end
|
|
31
34
|
|
|
35
|
+
def retain_until(timestamp)
|
|
36
|
+
hour = timestamp.utc.beginning_of_hour
|
|
37
|
+
hour + limit.send(unit) if hour == hour.send(:"beginning_of_#{unit.to_s.singularize}")
|
|
38
|
+
end
|
|
39
|
+
|
|
32
40
|
private
|
|
33
41
|
|
|
34
42
|
def ago count
|
|
@@ -38,4 +46,3 @@ module Bard
|
|
|
38
46
|
end
|
|
39
47
|
end
|
|
40
48
|
end
|
|
41
|
-
|
|
@@ -8,7 +8,8 @@ module Bard
|
|
|
8
8
|
class Backup
|
|
9
9
|
class S3Destination < Destination
|
|
10
10
|
def call
|
|
11
|
-
|
|
11
|
+
upload_tree = s3_tree.object_lock_enabled? ? s3_tree.with(retain_until: Deleter.retain_until(now)) : s3_tree
|
|
12
|
+
strategy.call(upload_tree, now)
|
|
12
13
|
Deleter.new(s3_tree, now).call
|
|
13
14
|
Bard::Backup.new(timestamp: now, destinations: [info])
|
|
14
15
|
end
|
|
@@ -96,8 +96,8 @@ module Bard
|
|
|
96
96
|
data_paths.each do |data_path|
|
|
97
97
|
Dir.glob("#{data_path}/**/*").each do |file|
|
|
98
98
|
next unless File.file?(file)
|
|
99
|
-
# Rails keeps
|
|
100
|
-
next if File.basename(file).match?(
|
|
99
|
+
# Rails keeps SQLite databases in storage/: the dump backs up the app's, the rest are runtime state
|
|
100
|
+
next if File.basename(file).match?(/\.sqlite3(-wal|-shm|-journal)?\z/)
|
|
101
101
|
stat = File.stat(file)
|
|
102
102
|
result[file] = { mtime: stat.mtime.to_f, size: stat.size }
|
|
103
103
|
end
|
|
@@ -1,13 +1,16 @@
|
|
|
1
1
|
require "backhoe"
|
|
2
|
+
require "bard/config"
|
|
3
|
+
require "tmpdir"
|
|
2
4
|
|
|
3
5
|
module Bard
|
|
4
6
|
class Backup
|
|
5
7
|
class LocalBackhoe
|
|
6
8
|
def self.call s3_tree, now
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
9
|
+
Dir.mktmpdir("#{Bard::Config.current.project_name}-backup-") do |dir|
|
|
10
|
+
path = File.join(dir, "#{now.iso8601}.sql.gz")
|
|
11
|
+
Backhoe.dump path
|
|
12
|
+
s3_tree.mv path
|
|
13
|
+
end
|
|
11
14
|
end
|
|
12
15
|
end
|
|
13
16
|
end
|
|
@@ -0,0 +1,141 @@
|
|
|
1
|
+
require "backhoe"
|
|
2
|
+
require "tmpdir"
|
|
3
|
+
require "zlib"
|
|
4
|
+
require "active_support/hash_with_indifferent_access"
|
|
5
|
+
require "bard/backup/finder"
|
|
6
|
+
|
|
7
|
+
module Bard
|
|
8
|
+
class Backup
|
|
9
|
+
# Restores the latest database backup into a scratch database beside the live one, checks
|
|
10
|
+
# that it holds real data, and drops it. Raises Failed with every problem found.
|
|
11
|
+
class RestoreCheck < Struct.new(:destinations, :database, keyword_init: true)
|
|
12
|
+
class Failed < StandardError; end
|
|
13
|
+
|
|
14
|
+
def self.call(...)
|
|
15
|
+
new(...).call
|
|
16
|
+
end
|
|
17
|
+
|
|
18
|
+
def call
|
|
19
|
+
found = Finder.new(destinations).find
|
|
20
|
+
backup = Bard::Backup.new(timestamp: found[:timestamp], destinations: [found[:destination].info])
|
|
21
|
+
label = "#{backup.timestamp.iso8601} (#{found[:destination].info[:name]})"
|
|
22
|
+
|
|
23
|
+
Dir.mktmpdir("bard-backup-restore-check-") do |dir|
|
|
24
|
+
path = File.join(dir, File.basename(found[:filename]))
|
|
25
|
+
File.binwrite(path, found[:destination].s3_tree.get(found[:filename]))
|
|
26
|
+
scratch = ScratchDatabase.new(live_config, dir)
|
|
27
|
+
|
|
28
|
+
fail! label, DumpIntegrity.new(path, scratch.database).problems
|
|
29
|
+
row_counts = scratch.load(path)
|
|
30
|
+
fail! label, sanity_problems(row_counts)
|
|
31
|
+
Result.new(backup:, row_counts:)
|
|
32
|
+
end
|
|
33
|
+
end
|
|
34
|
+
|
|
35
|
+
class Result < Data.define(:backup, :row_counts)
|
|
36
|
+
def to_s
|
|
37
|
+
"Restore check passed for backup #{backup.timestamp.iso8601}: #{row_counts.size} tables, #{row_counts.values.sum} rows"
|
|
38
|
+
end
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
private
|
|
42
|
+
|
|
43
|
+
def live_config
|
|
44
|
+
ActiveSupport::HashWithIndifferentAccess.new(database || Backhoe::Database.new.config)
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
def fail!(label, problems)
|
|
48
|
+
raise Failed, "Restore check failed for backup #{label}: #{problems.join("; ")}" if problems.any?
|
|
49
|
+
end
|
|
50
|
+
|
|
51
|
+
def sanity_problems(row_counts)
|
|
52
|
+
return ["the restored database has no tables"] if row_counts.empty?
|
|
53
|
+
problems = []
|
|
54
|
+
problems << "schema_migrations is missing or empty" unless row_counts["schema_migrations"].to_i.positive?
|
|
55
|
+
problems << "every table is empty" if row_counts.except("schema_migrations", "ar_internal_metadata").values.all?(&:zero?)
|
|
56
|
+
problems
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
# Catches the corruption seen in production: zero-filled heads, CRC errors, another
|
|
60
|
+
# project's gzip appended (which zcat and GzipReader silently ignore), and cut-short dumps.
|
|
61
|
+
class DumpIntegrity < Struct.new(:path, :database)
|
|
62
|
+
TAIL_BYTES = 64 * 1024
|
|
63
|
+
|
|
64
|
+
def problems
|
|
65
|
+
head, tail, trailing = read
|
|
66
|
+
return ["the dump has #{trailing} unexpected bytes after its gzip stream"] if trailing.positive?
|
|
67
|
+
return ["the dump is empty"] if head.empty?
|
|
68
|
+
|
|
69
|
+
if database.mysql? && !tail.match?(/^-- Dump completed/)
|
|
70
|
+
['the MySQL dump has no "-- Dump completed" line, so it was cut short']
|
|
71
|
+
elsif database.postgresql? && !tail.match?(/^-- PostgreSQL database dump complete/)
|
|
72
|
+
['the PostgreSQL dump has no "-- PostgreSQL database dump complete" line, so it was cut short']
|
|
73
|
+
elsif database.sqlite? && !head.start_with?("SQLite format 3\0")
|
|
74
|
+
["the dump is not an SQLite database"]
|
|
75
|
+
else
|
|
76
|
+
[]
|
|
77
|
+
end
|
|
78
|
+
rescue Zlib::Error => error
|
|
79
|
+
["the dump is not a valid gzip file: #{error.message}"]
|
|
80
|
+
end
|
|
81
|
+
|
|
82
|
+
private
|
|
83
|
+
|
|
84
|
+
def read
|
|
85
|
+
File.open(path, "rb") do |file|
|
|
86
|
+
gzip = Zlib::GzipReader.new(file)
|
|
87
|
+
head = gzip.read(16) || "".b
|
|
88
|
+
tail = head.dup
|
|
89
|
+
while (chunk = gzip.read(1024 * 1024))
|
|
90
|
+
tail = (tail + chunk)[-TAIL_BYTES..] || tail + chunk
|
|
91
|
+
end
|
|
92
|
+
trailing = gzip.unused.to_s.bytesize + (file.size - file.pos)
|
|
93
|
+
gzip.finish
|
|
94
|
+
[head, tail, trailing]
|
|
95
|
+
end
|
|
96
|
+
end
|
|
97
|
+
end
|
|
98
|
+
|
|
99
|
+
class ScratchDatabase
|
|
100
|
+
attr_reader :config, :database
|
|
101
|
+
|
|
102
|
+
def initialize(live_config, dir)
|
|
103
|
+
live = Backhoe::Database.new(live_config)
|
|
104
|
+
name = live.sqlite? ? File.join(dir, "restore_check.sqlite3") : "#{live.name}_restore_check"
|
|
105
|
+
@config = live_config.merge("database" => name)
|
|
106
|
+
@database = Backhoe::Database.new(config)
|
|
107
|
+
end
|
|
108
|
+
|
|
109
|
+
def load(path)
|
|
110
|
+
require "bard/backup/restore_check/scratch_record"
|
|
111
|
+
Backhoe::Load.new(database, path, true).call
|
|
112
|
+
row_counts
|
|
113
|
+
ensure
|
|
114
|
+
drop
|
|
115
|
+
end
|
|
116
|
+
|
|
117
|
+
private
|
|
118
|
+
|
|
119
|
+
def row_counts
|
|
120
|
+
ScratchRecord.establish_connection(config.to_h)
|
|
121
|
+
ScratchRecord.connection_pool.with_connection do |connection|
|
|
122
|
+
connection.tables.sort.to_h do |table|
|
|
123
|
+
[table, connection.select_value("SELECT COUNT(*) FROM #{connection.quote_table_name(table)}").to_i]
|
|
124
|
+
end
|
|
125
|
+
end
|
|
126
|
+
end
|
|
127
|
+
|
|
128
|
+
# Not DatabaseTasks.drop: it would point ActiveRecord::Base at the scratch database.
|
|
129
|
+
def drop
|
|
130
|
+
ScratchRecord.remove_connection
|
|
131
|
+
return if database.sqlite?
|
|
132
|
+
server = database.postgresql? ? config.merge("database" => "postgres") : config.except("database")
|
|
133
|
+
ScratchRecord.establish_connection(server.to_h)
|
|
134
|
+
ScratchRecord.connection_pool.with_connection { |connection| connection.drop_database(config["database"]) }
|
|
135
|
+
ensure
|
|
136
|
+
ScratchRecord.remove_connection
|
|
137
|
+
end
|
|
138
|
+
end
|
|
139
|
+
end
|
|
140
|
+
end
|
|
141
|
+
end
|
data/lib/bard/backup/s3_tree.rb
CHANGED
|
@@ -5,11 +5,12 @@ require "bard/backup/encryptor"
|
|
|
5
5
|
|
|
6
6
|
module Bard
|
|
7
7
|
class Backup
|
|
8
|
-
class S3Tree < Data.define(:endpoint, :path, :access_key_id, :secret_access_key, :region, :session_token, :encryption_key)
|
|
8
|
+
class S3Tree < Data.define(:endpoint, :path, :access_key_id, :secret_access_key, :region, :session_token, :encryption_key, :retain_until)
|
|
9
9
|
def initialize(**kwargs)
|
|
10
10
|
kwargs[:endpoint] ||= "https://s3.#{kwargs[:region]}.amazonaws.com"
|
|
11
11
|
kwargs[:session_token] ||= nil
|
|
12
12
|
kwargs[:encryption_key] ||= nil
|
|
13
|
+
kwargs[:retain_until] ||= nil
|
|
13
14
|
super
|
|
14
15
|
end
|
|
15
16
|
|
|
@@ -62,11 +63,15 @@ module Bard
|
|
|
62
63
|
|
|
63
64
|
def put_body(remote_key, body)
|
|
64
65
|
body = encryptor.encrypt(body) if encryptor
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
66
|
+
params = { bucket: bucket_name, key: [folder_prefix, remote_key].compact.join("/"), body: body }
|
|
67
|
+
params.merge!(object_lock_mode: "GOVERNANCE", object_lock_retain_until_date: retain_until) if retain_until
|
|
68
|
+
client.put_object(params)
|
|
69
|
+
end
|
|
70
|
+
|
|
71
|
+
def object_lock_enabled?
|
|
72
|
+
client.get_object_lock_configuration(bucket: bucket_name).object_lock_configuration.object_lock_enabled == "Enabled"
|
|
73
|
+
rescue Aws::S3::Errors::ObjectLockConfigurationNotFoundError
|
|
74
|
+
false
|
|
70
75
|
end
|
|
71
76
|
|
|
72
77
|
def mv(local_path)
|
data/lib/bard/backup/tasks.rake
CHANGED
|
@@ -9,6 +9,7 @@ namespace :bard do
|
|
|
9
9
|
end
|
|
10
10
|
|
|
11
11
|
Bard::Backup.create!(destinations, **Bard::Backup::RailsCredentials.find)
|
|
12
|
+
Bard::Backup::UploadCoverage.check!
|
|
12
13
|
end
|
|
13
14
|
|
|
14
15
|
namespace :backup do
|
|
@@ -17,6 +18,7 @@ namespace :bard do
|
|
|
17
18
|
require "bard/backup"
|
|
18
19
|
require "bard/backup/rails_credentials"
|
|
19
20
|
Bard::Backup::FileTree.create!(**Bard::Backup::RailsCredentials.find)
|
|
21
|
+
Bard::Backup::UploadCoverage.check!
|
|
20
22
|
end
|
|
21
23
|
|
|
22
24
|
desc "Encrypt the backups and data files stored before encryption was enabled"
|
|
@@ -34,6 +36,12 @@ namespace :bard do
|
|
|
34
36
|
end
|
|
35
37
|
end
|
|
36
38
|
|
|
39
|
+
desc "Restore the latest database backup into a scratch database and check that it holds data"
|
|
40
|
+
task :check => :environment do
|
|
41
|
+
require "bard/backup"
|
|
42
|
+
puts Bard::Backup.check_restore!(destinations: Bard::Config.current.backup.destinations)
|
|
43
|
+
end
|
|
44
|
+
|
|
37
45
|
desc "Restore the database backup nearest a timestamp (no timestamp lists available backups)"
|
|
38
46
|
task :restore, [:at] => :environment do |_task, args|
|
|
39
47
|
require "bard/backup"
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
require "pathname"
|
|
2
|
+
|
|
3
|
+
module Bard
|
|
4
|
+
class Backup
|
|
5
|
+
class UncoveredUploads < StandardError; end
|
|
6
|
+
|
|
7
|
+
# Uploads on local disk are only backed up when bard.rb declares their directory as data.
|
|
8
|
+
class UploadCoverage < Struct.new(:service, :data_paths)
|
|
9
|
+
def self.check!(service = (ActiveStorage::Blob.service if defined?(ActiveStorage::Blob)), data_paths = Bard::Config.current.data)
|
|
10
|
+
new(service, data_paths).check!
|
|
11
|
+
end
|
|
12
|
+
|
|
13
|
+
def check!
|
|
14
|
+
return if service.nil? || covered?(service)
|
|
15
|
+
roots = disk_roots(service).map { |root| display(root) }
|
|
16
|
+
raise UncoveredUploads, "Active Storage keeps uploads on this server in #{roots.join(" and ")}, " \
|
|
17
|
+
"which bard.rb does not declare as data, so they are not backed up. Add it to `data` in bard.rb."
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
private
|
|
21
|
+
|
|
22
|
+
def covered?(service)
|
|
23
|
+
if disk?(service)
|
|
24
|
+
root = File.expand_path(service.root)
|
|
25
|
+
data_paths.any? { |path| root == File.expand_path(path) || root.start_with?("#{File.expand_path(path)}/") }
|
|
26
|
+
elsif mirror?(service)
|
|
27
|
+
[service.primary, *service.mirrors].any? { |member| covered?(member) }
|
|
28
|
+
else
|
|
29
|
+
true
|
|
30
|
+
end
|
|
31
|
+
end
|
|
32
|
+
|
|
33
|
+
def disk_roots(service)
|
|
34
|
+
return [service.root] if disk?(service)
|
|
35
|
+
return [service.primary, *service.mirrors].flat_map { |member| disk_roots(member) } if mirror?(service)
|
|
36
|
+
[]
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
def disk?(service)
|
|
40
|
+
defined?(ActiveStorage::Service::DiskService) && service.is_a?(ActiveStorage::Service::DiskService)
|
|
41
|
+
end
|
|
42
|
+
|
|
43
|
+
def mirror?(service)
|
|
44
|
+
defined?(ActiveStorage::Service::MirrorService) && service.is_a?(ActiveStorage::Service::MirrorService)
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
def display(root)
|
|
48
|
+
relative = Pathname(File.expand_path(root)).relative_path_from(Dir.pwd).to_s
|
|
49
|
+
relative.start_with?("..") ? root.to_s : relative
|
|
50
|
+
end
|
|
51
|
+
end
|
|
52
|
+
end
|
|
53
|
+
end
|
data/lib/bard/backup/version.rb
CHANGED
data/lib/bard/backup.rb
CHANGED
|
@@ -4,7 +4,9 @@ require "bard/backup/database"
|
|
|
4
4
|
require "bard/backup/file_tree"
|
|
5
5
|
require "bard/backup/finder"
|
|
6
6
|
require "bard/backup/restore"
|
|
7
|
+
require "bard/backup/restore_check"
|
|
7
8
|
require "bard/backup/rekey"
|
|
9
|
+
require "bard/backup/upload_coverage"
|
|
8
10
|
require "bard/backup/railtie" if defined?(Rails)
|
|
9
11
|
|
|
10
12
|
module Bard
|
|
@@ -47,6 +49,10 @@ module Bard
|
|
|
47
49
|
Restore.call(at: at, drop_and_create: drop_and_create, destinations: destinations)
|
|
48
50
|
end
|
|
49
51
|
|
|
52
|
+
def self.check_restore!(destinations: nil, database: nil)
|
|
53
|
+
RestoreCheck.call(destinations:, database:)
|
|
54
|
+
end
|
|
55
|
+
|
|
50
56
|
attr_reader :timestamp, :size, :destinations
|
|
51
57
|
|
|
52
58
|
def initialize(timestamp:, size: nil, destinations: [])
|
metadata
CHANGED
|
@@ -1,13 +1,13 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: bard-backup
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.17.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Micah Geisel
|
|
8
8
|
bindir: exe
|
|
9
9
|
cert_chain: []
|
|
10
|
-
date:
|
|
10
|
+
date: 1980-01-02 00:00:00.000000000 Z
|
|
11
11
|
dependencies:
|
|
12
12
|
- !ruby/object:Gem::Dependency
|
|
13
13
|
name: backhoe
|
|
@@ -15,14 +15,14 @@ dependencies:
|
|
|
15
15
|
requirements:
|
|
16
16
|
- - ">="
|
|
17
17
|
- !ruby/object:Gem::Version
|
|
18
|
-
version: '0'
|
|
18
|
+
version: '0.11'
|
|
19
19
|
type: :runtime
|
|
20
20
|
prerelease: false
|
|
21
21
|
version_requirements: !ruby/object:Gem::Requirement
|
|
22
22
|
requirements:
|
|
23
23
|
- - ">="
|
|
24
24
|
- !ruby/object:Gem::Version
|
|
25
|
-
version: '0'
|
|
25
|
+
version: '0.11'
|
|
26
26
|
- !ruby/object:Gem::Dependency
|
|
27
27
|
name: aws-sdk-s3
|
|
28
28
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -109,8 +109,11 @@ files:
|
|
|
109
109
|
- lib/bard/backup/railtie.rb
|
|
110
110
|
- lib/bard/backup/rekey.rb
|
|
111
111
|
- lib/bard/backup/restore.rb
|
|
112
|
+
- lib/bard/backup/restore_check.rb
|
|
113
|
+
- lib/bard/backup/restore_check/scratch_record.rb
|
|
112
114
|
- lib/bard/backup/s3_tree.rb
|
|
113
115
|
- lib/bard/backup/tasks.rake
|
|
116
|
+
- lib/bard/backup/upload_coverage.rb
|
|
114
117
|
- lib/bard/backup/version.rb
|
|
115
118
|
- lib/bard/plugins/backup.rb
|
|
116
119
|
- lib/bard/plugins/backup_restore.rb
|
|
@@ -134,7 +137,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
134
137
|
- !ruby/object:Gem::Version
|
|
135
138
|
version: '0'
|
|
136
139
|
requirements: []
|
|
137
|
-
rubygems_version:
|
|
140
|
+
rubygems_version: 4.0.20
|
|
138
141
|
specification_version: 4
|
|
139
142
|
summary: Provides automated db backups for bard projects
|
|
140
143
|
test_files: []
|