pgbackups-archive 0.3.0 → 1.0.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/MIT-LICENSE +1 -1
- data/README.md +35 -37
- data/lib/pgbackups-archive.rb +1 -1
- data/lib/pgbackups-archive/job.rb +109 -0
- data/lib/pgbackups-archive/version.rb +1 -1
- data/lib/tasks/pgbackups_archive.rake +2 -2
- data/test/lib/pgbackups-archive/job_test.rb +131 -0
- data/test/lib/pgbackups-archive/storage_test.rb +16 -16
- metadata +39 -14
- data/lib/pgbackups-archive/heroku/client/pgbackups_archive.rb +0 -77
- data/test/lib/pgbackups-archive/heroku/client/pgbackups_archive_test.rb +0 -181
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 8c51d872d12dc01426e3a9185e63a118b204ecdd
|
4
|
+
data.tar.gz: 92b49a4dcbb9ef8eae5d2ed1f6a9892bfa01f52b
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 409279267d694930ee6e35452bdfdbb23cf5955769be967ea9fef96fe133683306bf541623a969e65823245559c84ae448b52466f8493371017228ffc5f817a7
|
7
|
+
data.tar.gz: 021689847fb1cbade0f0c45829d1a351ed2d89dd6e10df572ec7e5fd09dc10ab6418bca37fd092c591fbbe9e3ac5286bd870b1b198406228f9d357967d291584
|
data/MIT-LICENSE
CHANGED
data/README.md
CHANGED
@@ -3,66 +3,64 @@
|
|
3
3
|
[![Gem Version](https://badge.fury.io/rb/pgbackups-archive.svg)](http://badge.fury.io/rb/pgbackups-archive)
|
4
4
|
[![Code Climate](https://codeclimate.com/github/kjohnston/pgbackups-archive/badges/gpa.svg)](https://codeclimate.com/github/kjohnston/pgbackups-archive)
|
5
5
|
|
6
|
-
A means of automating Heroku
|
6
|
+
A means of automating Heroku PGBackups and archiving them to Amazon S3.
|
7
|
+
|
8
|
+
## Upgrade Alert
|
9
|
+
|
10
|
+
As of v1.0.0, `pgbackups-archive` works with the new [Heroku PGBackups](https://devcenter.heroku.com/articles/heroku-postgres-backups) service, which replaced the older [PG Backups](https://devcenter.heroku.com/articles/pgbackups) add-on.
|
11
|
+
|
12
|
+
If you're still using an older version of `pgbackups-archive`, it's time to upgrade!
|
13
|
+
|
14
|
+
Read more about this transition in Heroku's offerings on the Heroku Blog: [PG Backups Levels Up](https://blog.heroku.com/archives/2015/3/11/pgbackups-levels-up)
|
15
|
+
|
16
|
+
Please note that the environment variables that need to be defined have changed with
|
17
|
+
v1.0.0.
|
7
18
|
|
8
19
|
## Overview
|
9
20
|
|
10
|
-
The `pgbackups:archive` rake task this gem provides will capture a
|
21
|
+
The `pgbackups:archive` rake task this gem provides will capture a Heroku PGBackup, wait for it to complete, then store it within the Amazon S3 bucket you specify. This rake task can be scheduled via the Heroku Scheduler, thus producing automated, offsite, backups.
|
22
|
+
|
23
|
+
This gem doesn't interfere with or utilze automated backups, so feel free to schedule those with the `pg:backups schedule` command as you desire.
|
11
24
|
|
12
|
-
|
25
|
+
You can configure how many manual backups (created by you or this gem) you'd like to keep at the Heroku PGBackups level to ensure there is always space to capture a new backup.
|
13
26
|
|
14
27
|
You can configure retention settings at the Amazon S3 bucket level from within the AWS Console if you like.
|
15
28
|
|
16
29
|
## Use
|
17
30
|
|
18
|
-
###
|
19
|
-
|
20
|
-
#### Option 1 - Add `pgbackups-archive` to your existing application
|
21
|
-
|
22
|
-
Recommended for apps with light memory usage.
|
31
|
+
### Install the gem
|
23
32
|
|
24
33
|
Add the gem to your Gemfile and bundle:
|
25
34
|
|
26
35
|
gem "pgbackups-archive"
|
27
36
|
bundle install
|
28
37
|
|
29
|
-
|
30
|
-
|
31
|
-
Recommended for apps with heavier memory usage.
|
32
|
-
|
33
|
-
* Create a new repo with `pgackups-archive` added to that app's Gemfile and push
|
34
|
-
it to a new Heroku app.
|
35
|
-
* Ensure the `PGBACKUPS_DATABASE_URL` environment variable you set for your
|
36
|
-
backup app points to your main app's `DATABASE_URL`, or other follower URL, so
|
37
|
-
that `pgbackups-archive` in your backup app knows to backup your real app's
|
38
|
-
database.
|
38
|
+
### Install Heroku Scheduler add-on
|
39
39
|
|
40
|
-
|
41
|
-
will utilize a certain amount of memory beyond what an instance of your
|
42
|
-
application uses and if you're close to the threshold of your Dyno size as it
|
43
|
-
is, this increment could put the instance over the limit and cause it to
|
44
|
-
encounter [memory allocation errors](https://devcenter.heroku.com/articles/error-codes#r14-memory-quota-exceeded).
|
45
|
-
By running a dedicated Heroku app to run
|
46
|
-
`pgbackups-archive` the task will have ample room at the 1X Dyno level to stream
|
47
|
-
the backup files.
|
40
|
+
heroku addons:add scheduler:standard
|
48
41
|
|
49
|
-
###
|
42
|
+
### Setup an AWS IAM user, S3 bucket and policy
|
50
43
|
|
51
|
-
|
52
|
-
heroku addons:add scheduler:standard
|
44
|
+
A good security measure would be to use a dedicated set of AWS credentials with a security policy only allowing access to the bucket you're specifying. See this Pro Tip on [Assigning an AWS IAM user access to a single S3 bucket](http://coderwall.com/p/dwhlma).
|
53
45
|
|
54
|
-
### Apply
|
46
|
+
### Apply Environment Variables
|
55
47
|
|
48
|
+
# Required
|
49
|
+
heroku config:add HEROKU_API_KEY="collaborator-api-key"
|
50
|
+
heroku config:add PGBACKUPS_APP="myapp"
|
56
51
|
heroku config:add PGBACKUPS_AWS_ACCESS_KEY_ID="XXX"
|
57
52
|
heroku config:add PGBACKUPS_AWS_SECRET_ACCESS_KEY="YYY"
|
58
53
|
heroku config:add PGBACKUPS_BUCKET="myapp-backups"
|
59
54
|
heroku config:add PGBACKUPS_REGION="us-west-2"
|
60
|
-
heroku config:add PGBACKUPS_DATABASE_URL="your main app's DATABASE_URL or other follower URL here"
|
61
55
|
|
62
|
-
|
63
|
-
|
64
|
-
|
65
|
-
|
56
|
+
# Optional: If you wish to backup a database other than the one that
|
57
|
+
# DATABASE_URL points to, set this to the name of the variable for that
|
58
|
+
# database (useful for follower databases).
|
59
|
+
heroku config:add PGBACKUPS_DATABASE="HEROKU_POSTGRESQL_BLACK_URL"
|
60
|
+
|
61
|
+
# Optional: If you wish to customize the number of manual backups kept at
|
62
|
+
# the Heroku PGBackups level, set this.
|
63
|
+
heroku config:add PGBACKUPS_KEEP="30"
|
66
64
|
|
67
65
|
### Add the rake task to scheduler
|
68
66
|
|
@@ -92,11 +90,11 @@ I shouldn't have to say this, but I will. Your backups are your responsibility.
|
|
92
90
|
|
93
91
|
## Contributing
|
94
92
|
|
95
|
-
1.
|
93
|
+
1. Fork it
|
96
94
|
2. Create your feature branch (`git checkout -b my-new-feature`)
|
97
95
|
3. Commit your changes (`git commit -am 'Added some feature'`)
|
98
96
|
4. Push to the branch (`git push origin my-new-feature`)
|
99
|
-
5.
|
97
|
+
5. Create a Pull Request
|
100
98
|
|
101
99
|
## Contributors
|
102
100
|
|
data/lib/pgbackups-archive.rb
CHANGED
@@ -0,0 +1,109 @@
|
|
1
|
+
require "heroku/command/pg"
|
2
|
+
require "heroku/command/pg_backups"
|
3
|
+
require "heroku/api"
|
4
|
+
require "tmpdir"
|
5
|
+
|
6
|
+
class PgbackupsArchive::Job
|
7
|
+
|
8
|
+
attr_reader :client
|
9
|
+
attr_accessor :backup_url, :created_at
|
10
|
+
|
11
|
+
def self.call
|
12
|
+
new.call
|
13
|
+
end
|
14
|
+
|
15
|
+
def initialize(attrs={})
|
16
|
+
Heroku::Command.load
|
17
|
+
@client = Heroku::Command::Pg.new([], :app => ENV["PGBACKUPS_APP"])
|
18
|
+
end
|
19
|
+
|
20
|
+
def call
|
21
|
+
expire
|
22
|
+
capture
|
23
|
+
download
|
24
|
+
archive
|
25
|
+
delete
|
26
|
+
end
|
27
|
+
|
28
|
+
def archive
|
29
|
+
if PgbackupsArchive::Storage.new(key, file).store
|
30
|
+
client.display "Backup archived"
|
31
|
+
end
|
32
|
+
end
|
33
|
+
|
34
|
+
def capture
|
35
|
+
attachment = client.send(:generate_resolver).resolve(database)
|
36
|
+
backup = client.send(:hpg_client, attachment).backups_capture
|
37
|
+
client.send(:poll_transfer, "backup", backup[:uuid])
|
38
|
+
|
39
|
+
self.created_at = backup[:created_at]
|
40
|
+
|
41
|
+
self.backup_url = Heroku::Client::HerokuPostgresqlApp
|
42
|
+
.new(ENV["PGBACKUPS_APP"]).transfers_public_url(backup[:num])[:url]
|
43
|
+
end
|
44
|
+
|
45
|
+
def delete
|
46
|
+
File.delete(temp_file)
|
47
|
+
end
|
48
|
+
|
49
|
+
def download
|
50
|
+
File.open(temp_file, "wb") do |output|
|
51
|
+
streamer = lambda do |chunk, remaining_bytes, total_bytes|
|
52
|
+
output.write chunk
|
53
|
+
end
|
54
|
+
|
55
|
+
# https://github.com/excon/excon/issues/475
|
56
|
+
Excon.get backup_url,
|
57
|
+
:response_block => streamer,
|
58
|
+
:omit_default_port => true
|
59
|
+
end
|
60
|
+
end
|
61
|
+
|
62
|
+
def expire
|
63
|
+
transfers = client.send(:hpg_app_client, ENV["PGBACKUPS_APP"]).transfers
|
64
|
+
.select { |b| b[:from_type] == "pg_dump" && b[:to_type] == "gof3r" }
|
65
|
+
.sort_by { |b| b[:created_at] }
|
66
|
+
|
67
|
+
if transfers.size > pgbackups_to_keep
|
68
|
+
backup_id = "b%03d" % transfers.first[:num]
|
69
|
+
backup_num = client.send(:backup_num, backup_id)
|
70
|
+
|
71
|
+
expire_backup(backup_num)
|
72
|
+
|
73
|
+
client.display "Backup #{backup_id} expired"
|
74
|
+
end
|
75
|
+
end
|
76
|
+
|
77
|
+
private
|
78
|
+
|
79
|
+
def expire_backup(backup_num)
|
80
|
+
client.send(:hpg_app_client, ENV["PGBACKUPS_APP"])
|
81
|
+
.transfers_delete(backup_num)
|
82
|
+
end
|
83
|
+
|
84
|
+
def database
|
85
|
+
ENV["PGBACKUPS_DATABASE"] || "DATABASE_URL"
|
86
|
+
end
|
87
|
+
|
88
|
+
def environment
|
89
|
+
defined?(Rails) ? Rails.env : nil
|
90
|
+
end
|
91
|
+
|
92
|
+
def file
|
93
|
+
File.open(temp_file, "r")
|
94
|
+
end
|
95
|
+
|
96
|
+
def key
|
97
|
+
timestamp = created_at.gsub(/\/|\:|\.|\s/, "-").concat(".dump")
|
98
|
+
["pgbackups", environment, timestamp].compact.join("/")
|
99
|
+
end
|
100
|
+
|
101
|
+
def pgbackups_to_keep
|
102
|
+
var = ENV["PGBACKUPS_KEEP"] ? var.to_i : 30
|
103
|
+
end
|
104
|
+
|
105
|
+
def temp_file
|
106
|
+
"#{Dir.tmpdir}/pgbackup"
|
107
|
+
end
|
108
|
+
|
109
|
+
end
|
@@ -0,0 +1,131 @@
|
|
1
|
+
require "test_helper"
|
2
|
+
|
3
|
+
describe PgbackupsArchive::Job do
|
4
|
+
|
5
|
+
describe ".call" do
|
6
|
+
it { PgbackupsArchive::Job.must_respond_to(:call) }
|
7
|
+
end
|
8
|
+
|
9
|
+
describe "instance methods" do
|
10
|
+
before do
|
11
|
+
@app = "foobar"
|
12
|
+
ENV["PGBACKUPS_APP"] = @app
|
13
|
+
@job = PgbackupsArchive::Job.new
|
14
|
+
@client = @job.client
|
15
|
+
@backup_url = "https://raw.githubusercontent.com/kjohnston/pgbackups-archive/master/pgbackups-archive.gemspec"
|
16
|
+
@created_at = Time.now
|
17
|
+
end
|
18
|
+
|
19
|
+
describe "#initialize" do
|
20
|
+
it do
|
21
|
+
@client.must_be_kind_of(Heroku::Command::Pg)
|
22
|
+
@client.options[:app].must_equal @app
|
23
|
+
end
|
24
|
+
end
|
25
|
+
|
26
|
+
describe "#call" do
|
27
|
+
before do
|
28
|
+
@job.expects(:expire)
|
29
|
+
@job.expects(:capture)
|
30
|
+
@job.expects(:download)
|
31
|
+
@job.expects(:archive)
|
32
|
+
@job.expects(:delete)
|
33
|
+
end
|
34
|
+
|
35
|
+
it { @job.call }
|
36
|
+
end
|
37
|
+
|
38
|
+
describe "#archive" do
|
39
|
+
before do
|
40
|
+
@key = "some-key"
|
41
|
+
@file = "some-file"
|
42
|
+
@job.stubs(:key).returns(@key)
|
43
|
+
@job.stubs(:file).returns(@file)
|
44
|
+
|
45
|
+
PgbackupsArchive::Storage.expects(:new).with(@key, @file)
|
46
|
+
.returns(mock(store: true))
|
47
|
+
|
48
|
+
@client.expects(:display)
|
49
|
+
end
|
50
|
+
|
51
|
+
it { @job.archive }
|
52
|
+
end
|
53
|
+
|
54
|
+
describe "#capture" do
|
55
|
+
before do
|
56
|
+
@client.stubs(:generate_resolver).returns(mock(:resolve))
|
57
|
+
@client.stubs(:hpg_client)
|
58
|
+
.returns(
|
59
|
+
mock(backups_capture: {
|
60
|
+
uuid: "baz",
|
61
|
+
num: "10",
|
62
|
+
created_at: @created_at }
|
63
|
+
)
|
64
|
+
)
|
65
|
+
@client.stubs(:poll_transfer)
|
66
|
+
|
67
|
+
Heroku::Client::HerokuPostgresqlApp.expects(:new).with(@app)
|
68
|
+
.returns(mock(transfers_public_url: { url: @backup_url }))
|
69
|
+
end
|
70
|
+
|
71
|
+
it do
|
72
|
+
@job.capture
|
73
|
+
@job.created_at.must_equal @created_at
|
74
|
+
@job.backup_url.must_equal @backup_url
|
75
|
+
end
|
76
|
+
end
|
77
|
+
|
78
|
+
describe "#delete" do
|
79
|
+
it do
|
80
|
+
@temp_file = @job.send(:temp_file)
|
81
|
+
File.write(@temp_file, "content")
|
82
|
+
File.exist?(@temp_file).must_equal true
|
83
|
+
@job.delete
|
84
|
+
File.exist?(@temp_file).must_equal false
|
85
|
+
end
|
86
|
+
end
|
87
|
+
|
88
|
+
describe "#download" do
|
89
|
+
before { @job.backup_url = @backup_url }
|
90
|
+
|
91
|
+
it do
|
92
|
+
@job.download
|
93
|
+
@job.send(:file).read.must_match /Gem::Specification/
|
94
|
+
end
|
95
|
+
end
|
96
|
+
|
97
|
+
describe "#expire" do
|
98
|
+
before do
|
99
|
+
@transfers = [
|
100
|
+
{ from_type: "pg_dump", to_type: "gof3r", created_at: Date.today, num: 20 },
|
101
|
+
{ from_type: "pg_dump", to_type: "gof3r", created_at: Date.today-2, num: 17 },
|
102
|
+
{ from_type: "pg_dump", to_type: "gof3r", created_at: Date.today-1, num: 18 },
|
103
|
+
{ from_type: "pg_dump", to_type: "foo", created_at: Date.today-1, num: 19 }
|
104
|
+
]
|
105
|
+
|
106
|
+
@client.expects(:hpg_app_client).with(@app).returns(mock(transfers: @transfers))
|
107
|
+
end
|
108
|
+
|
109
|
+
describe "when slots are available" do
|
110
|
+
it "does not expire a backup" do
|
111
|
+
@job.expire
|
112
|
+
end
|
113
|
+
end
|
114
|
+
|
115
|
+
describe "when a slot needs to be freed" do
|
116
|
+
before do
|
117
|
+
ENV["PGBACKUPS_KEEP"] = "2"
|
118
|
+
@client.expects(:backup_num).with("b017").returns("017")
|
119
|
+
@job.expects(:expire_backup).with("017")
|
120
|
+
@client.expects(:display)
|
121
|
+
end
|
122
|
+
|
123
|
+
it "expires a backup" do
|
124
|
+
@job.expire
|
125
|
+
end
|
126
|
+
end
|
127
|
+
end
|
128
|
+
|
129
|
+
end
|
130
|
+
|
131
|
+
end
|
@@ -1,33 +1,33 @@
|
|
1
1
|
require "test_helper"
|
2
2
|
|
3
3
|
describe PgbackupsArchive::Storage do
|
4
|
-
let(:connection) {
|
5
|
-
Fog::Storage.new(
|
6
|
-
:provider => "AWS",
|
7
|
-
:aws_access_key_id => "XXX",
|
8
|
-
:aws_secret_access_key => "YYY")
|
9
|
-
}
|
10
|
-
let(:bucket) { connection.directories.create(:key => "someapp-backups") }
|
11
|
-
let(:key) { "pgbackups/test/2012-08-02-12-00-00.dump" }
|
12
|
-
let(:file) { "test" }
|
13
|
-
let(:storage) { PgbackupsArchive::Storage.new(key, file) }
|
14
|
-
|
15
4
|
before do
|
16
5
|
Fog.mock!
|
17
|
-
|
18
|
-
|
6
|
+
|
7
|
+
@connection = Fog::Storage.new(
|
8
|
+
provider: "AWS",
|
9
|
+
aws_access_key_id: "XXX",
|
10
|
+
aws_secret_access_key: "YYY"
|
11
|
+
)
|
12
|
+
@bucket = @connection.directories.create(key: "someapp-backups")
|
13
|
+
@key = "pgbackups/test/2012-08-02-12-00-00.dump"
|
14
|
+
@file = "test"
|
15
|
+
@storage = PgbackupsArchive::Storage.new(@key, @file)
|
16
|
+
|
17
|
+
@storage.stubs(:connection).returns(@connection)
|
18
|
+
@storage.stubs(:bucket).returns(@bucket)
|
19
19
|
end
|
20
20
|
|
21
21
|
it "should create a fog connection" do
|
22
|
-
storage.connection.class.must_equal Fog::Storage::AWS::Mock
|
22
|
+
@storage.connection.class.must_equal Fog::Storage::AWS::Mock
|
23
23
|
end
|
24
24
|
|
25
25
|
it "should create a fog directory" do
|
26
|
-
storage.bucket.class.must_equal Fog::Storage::AWS::Directory
|
26
|
+
@storage.bucket.class.must_equal Fog::Storage::AWS::Directory
|
27
27
|
end
|
28
28
|
|
29
29
|
it "should create a fog file" do
|
30
|
-
storage.store.class.must_equal Fog::Storage::AWS::File
|
30
|
+
@storage.store.class.must_equal Fog::Storage::AWS::File
|
31
31
|
end
|
32
32
|
|
33
33
|
end
|
metadata
CHANGED
@@ -1,43 +1,49 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: pgbackups-archive
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 1.0.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Kenny Johnston
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2015-
|
11
|
+
date: 2015-03-16 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
|
-
name:
|
14
|
+
name: heroku
|
15
15
|
requirement: !ruby/object:Gem::Requirement
|
16
16
|
requirements:
|
17
|
+
- - "~>"
|
18
|
+
- !ruby/object:Gem::Version
|
19
|
+
version: '3.28'
|
17
20
|
- - ">="
|
18
21
|
- !ruby/object:Gem::Version
|
19
|
-
version:
|
22
|
+
version: 3.28.6
|
20
23
|
type: :runtime
|
21
24
|
prerelease: false
|
22
25
|
version_requirements: !ruby/object:Gem::Requirement
|
23
26
|
requirements:
|
27
|
+
- - "~>"
|
28
|
+
- !ruby/object:Gem::Version
|
29
|
+
version: '3.28'
|
24
30
|
- - ">="
|
25
31
|
- !ruby/object:Gem::Version
|
26
|
-
version:
|
32
|
+
version: 3.28.6
|
27
33
|
- !ruby/object:Gem::Dependency
|
28
|
-
name:
|
34
|
+
name: fog-aws
|
29
35
|
requirement: !ruby/object:Gem::Requirement
|
30
36
|
requirements:
|
31
37
|
- - ">="
|
32
38
|
- !ruby/object:Gem::Version
|
33
|
-
version:
|
39
|
+
version: '0'
|
34
40
|
type: :runtime
|
35
41
|
prerelease: false
|
36
42
|
version_requirements: !ruby/object:Gem::Requirement
|
37
43
|
requirements:
|
38
44
|
- - ">="
|
39
45
|
- !ruby/object:Gem::Version
|
40
|
-
version:
|
46
|
+
version: '0'
|
41
47
|
- !ruby/object:Gem::Dependency
|
42
48
|
name: rake
|
43
49
|
requirement: !ruby/object:Gem::Requirement
|
@@ -66,6 +72,26 @@ dependencies:
|
|
66
72
|
- - ">="
|
67
73
|
- !ruby/object:Gem::Version
|
68
74
|
version: '0'
|
75
|
+
- !ruby/object:Gem::Dependency
|
76
|
+
name: guard
|
77
|
+
requirement: !ruby/object:Gem::Requirement
|
78
|
+
requirements:
|
79
|
+
- - "~>"
|
80
|
+
- !ruby/object:Gem::Version
|
81
|
+
version: '2.12'
|
82
|
+
- - "~>"
|
83
|
+
- !ruby/object:Gem::Version
|
84
|
+
version: 2.12.5
|
85
|
+
type: :development
|
86
|
+
prerelease: false
|
87
|
+
version_requirements: !ruby/object:Gem::Requirement
|
88
|
+
requirements:
|
89
|
+
- - "~>"
|
90
|
+
- !ruby/object:Gem::Version
|
91
|
+
version: '2.12'
|
92
|
+
- - "~>"
|
93
|
+
- !ruby/object:Gem::Version
|
94
|
+
version: 2.12.5
|
69
95
|
- !ruby/object:Gem::Dependency
|
70
96
|
name: guard-minitest
|
71
97
|
requirement: !ruby/object:Gem::Requirement
|
@@ -146,8 +172,7 @@ dependencies:
|
|
146
172
|
- - ">="
|
147
173
|
- !ruby/object:Gem::Version
|
148
174
|
version: 0.9.1
|
149
|
-
description: A means of automating Heroku
|
150
|
-
S3 via the fog gem.
|
175
|
+
description: A means of automating Heroku PGBackups and archiving them to Amazon S3.
|
151
176
|
email:
|
152
177
|
- kjohnston.ca@gmail.com
|
153
178
|
executables: []
|
@@ -158,7 +183,7 @@ files:
|
|
158
183
|
- README.md
|
159
184
|
- Rakefile
|
160
185
|
- lib/pgbackups-archive.rb
|
161
|
-
- lib/pgbackups-archive/
|
186
|
+
- lib/pgbackups-archive/job.rb
|
162
187
|
- lib/pgbackups-archive/railtie.rb
|
163
188
|
- lib/pgbackups-archive/storage.rb
|
164
189
|
- lib/pgbackups-archive/version.rb
|
@@ -166,7 +191,7 @@ files:
|
|
166
191
|
- lib/tasks/test.rake
|
167
192
|
- test/ci/before_script.sh
|
168
193
|
- test/ci/ci_runner.sh
|
169
|
-
- test/lib/pgbackups-archive/
|
194
|
+
- test/lib/pgbackups-archive/job_test.rb
|
170
195
|
- test/lib/pgbackups-archive/storage_test.rb
|
171
196
|
- test/support/mocha.rb
|
172
197
|
- test/test_helper.rb
|
@@ -193,11 +218,11 @@ rubyforge_project:
|
|
193
218
|
rubygems_version: 2.4.5
|
194
219
|
signing_key:
|
195
220
|
specification_version: 4
|
196
|
-
summary: Automates archival of Heroku
|
221
|
+
summary: Automates archival of Heroku PGBackups to S3
|
197
222
|
test_files:
|
198
223
|
- test/ci/before_script.sh
|
199
224
|
- test/ci/ci_runner.sh
|
200
|
-
- test/lib/pgbackups-archive/
|
225
|
+
- test/lib/pgbackups-archive/job_test.rb
|
201
226
|
- test/lib/pgbackups-archive/storage_test.rb
|
202
227
|
- test/support/mocha.rb
|
203
228
|
- test/test_helper.rb
|
@@ -1,77 +0,0 @@
|
|
1
|
-
require "heroku/client"
|
2
|
-
require "tmpdir"
|
3
|
-
|
4
|
-
class Heroku::Client::PgbackupsArchive
|
5
|
-
|
6
|
-
attr_reader :client, :pgbackup
|
7
|
-
|
8
|
-
def self.perform
|
9
|
-
backup = new
|
10
|
-
backup.capture
|
11
|
-
backup.download
|
12
|
-
backup.archive
|
13
|
-
backup.delete
|
14
|
-
end
|
15
|
-
|
16
|
-
def initialize(attrs={})
|
17
|
-
Heroku::Command.load
|
18
|
-
@client = Heroku::Client::Pgbackups.new pgbackups_url
|
19
|
-
@pgbackup = nil
|
20
|
-
end
|
21
|
-
|
22
|
-
def archive
|
23
|
-
PgbackupsArchive::Storage.new(key, file).store
|
24
|
-
end
|
25
|
-
|
26
|
-
def capture
|
27
|
-
@pgbackup = @client.create_transfer(database_url, database_url, nil,
|
28
|
-
"BACKUP", :expire => true)
|
29
|
-
|
30
|
-
until @pgbackup["finished_at"]
|
31
|
-
print "."
|
32
|
-
sleep 1
|
33
|
-
@pgbackup = @client.get_transfer @pgbackup["id"]
|
34
|
-
end
|
35
|
-
end
|
36
|
-
|
37
|
-
def delete
|
38
|
-
File.delete temp_file
|
39
|
-
end
|
40
|
-
|
41
|
-
def download
|
42
|
-
File.open(temp_file, "wb") do |output|
|
43
|
-
streamer = lambda do |chunk, remaining_bytes, total_bytes|
|
44
|
-
output.write chunk
|
45
|
-
end
|
46
|
-
Excon.get(@pgbackup["public_url"], :response_block => streamer)
|
47
|
-
end
|
48
|
-
end
|
49
|
-
|
50
|
-
private
|
51
|
-
|
52
|
-
def database_url
|
53
|
-
ENV["PGBACKUPS_DATABASE_URL"] || ENV["DATABASE_URL"]
|
54
|
-
end
|
55
|
-
|
56
|
-
def environment
|
57
|
-
defined?(Rails) ? Rails.env : nil
|
58
|
-
end
|
59
|
-
|
60
|
-
def file
|
61
|
-
File.open temp_file, "r"
|
62
|
-
end
|
63
|
-
|
64
|
-
def key
|
65
|
-
["pgbackups", environment, @pgbackup["finished_at"]
|
66
|
-
.gsub(/\/|\:|\.|\s/, "-").concat(".dump")].compact.join("/")
|
67
|
-
end
|
68
|
-
|
69
|
-
def pgbackups_url
|
70
|
-
ENV["PGBACKUPS_URL"]
|
71
|
-
end
|
72
|
-
|
73
|
-
def temp_file
|
74
|
-
"#{Dir.tmpdir}/#{URI(@pgbackup['public_url']).path.split('/').last}"
|
75
|
-
end
|
76
|
-
|
77
|
-
end
|
@@ -1,181 +0,0 @@
|
|
1
|
-
require "test_helper"
|
2
|
-
require "heroku/client"
|
3
|
-
|
4
|
-
describe Heroku::Client::PgbackupsArchive do
|
5
|
-
|
6
|
-
describe "#self.perform" do
|
7
|
-
before do
|
8
|
-
Heroku::Client::PgbackupsArchive.expects(:new).returns(
|
9
|
-
mock(
|
10
|
-
:capture => stub,
|
11
|
-
:download => stub,
|
12
|
-
:archive => stub,
|
13
|
-
:delete => stub
|
14
|
-
)
|
15
|
-
)
|
16
|
-
end
|
17
|
-
|
18
|
-
it { Heroku::Client::PgbackupsArchive.perform }
|
19
|
-
end
|
20
|
-
|
21
|
-
describe "An instance" do
|
22
|
-
let(:database_url) { "db_url" }
|
23
|
-
let(:pgbackups_url) { "https://ip:password@pgbackups.heroku.com/client" }
|
24
|
-
let(:backup) { Heroku::Client::PgbackupsArchive.new }
|
25
|
-
|
26
|
-
before do
|
27
|
-
ENV["PGBACKUPS_URL"] = pgbackups_url
|
28
|
-
end
|
29
|
-
|
30
|
-
describe "#initialize" do
|
31
|
-
it "should set client to a Heroku::Client::Pgbackups instance" do
|
32
|
-
backup.client.class.must_equal Heroku::Client::Pgbackups
|
33
|
-
end
|
34
|
-
end
|
35
|
-
|
36
|
-
describe "#archive" do
|
37
|
-
let(:key) { "some-key" }
|
38
|
-
let(:file) { "some-file" }
|
39
|
-
|
40
|
-
before do
|
41
|
-
backup.stubs(:key).returns(key)
|
42
|
-
backup.stubs(:file).returns(file)
|
43
|
-
|
44
|
-
PgbackupsArchive::Storage.expects(:new).with(key, file)
|
45
|
-
.returns(mock(:store => stub))
|
46
|
-
end
|
47
|
-
|
48
|
-
it "should use a storage instance to store the archive" do
|
49
|
-
backup.archive
|
50
|
-
end
|
51
|
-
end
|
52
|
-
|
53
|
-
describe "#capture" do
|
54
|
-
let(:pgbackup) { { "finished_at" => "some-timestamp" } }
|
55
|
-
|
56
|
-
before do
|
57
|
-
backup.stubs(:database_url).returns(database_url)
|
58
|
-
|
59
|
-
backup.client.expects(:create_transfer)
|
60
|
-
.with(database_url, database_url, nil, "BACKUP", :expire => true)
|
61
|
-
.returns(pgbackup)
|
62
|
-
end
|
63
|
-
|
64
|
-
it "uses the client to create a pgbackup" do
|
65
|
-
backup.capture
|
66
|
-
end
|
67
|
-
end
|
68
|
-
|
69
|
-
describe "#delete" do
|
70
|
-
let(:temp_file) { "temp-file" }
|
71
|
-
|
72
|
-
before do
|
73
|
-
backup.stubs(:temp_file).returns(temp_file)
|
74
|
-
File.expects(:delete).with(temp_file).returns(true)
|
75
|
-
end
|
76
|
-
|
77
|
-
it "should delete the temp file" do
|
78
|
-
backup.delete
|
79
|
-
end
|
80
|
-
end
|
81
|
-
|
82
|
-
describe "#download" do
|
83
|
-
let(:pgbackup) do
|
84
|
-
{
|
85
|
-
"public_url" => "https://raw.github.com/kjohnston/" +
|
86
|
-
"pgbackups-archive/master/pgbackups-archive.gemspec"
|
87
|
-
}
|
88
|
-
end
|
89
|
-
|
90
|
-
before do
|
91
|
-
backup.instance_eval do
|
92
|
-
@pgbackup = {
|
93
|
-
"public_url" => "https://raw.githubusercontent.com/kjohnston/pgbackups-archive/master/pgbackups-archive.gemspec"
|
94
|
-
}
|
95
|
-
end
|
96
|
-
backup.download
|
97
|
-
end
|
98
|
-
|
99
|
-
it "downloads the backup file" do
|
100
|
-
backup.send(:file).read.must_match /Gem::Specification/
|
101
|
-
end
|
102
|
-
|
103
|
-
after do
|
104
|
-
backup.delete
|
105
|
-
end
|
106
|
-
end
|
107
|
-
|
108
|
-
describe "#database_url" do
|
109
|
-
describe "when an alternate database to backup is not set" do
|
110
|
-
before do
|
111
|
-
ENV["PGBACKUPS_DATABASE_URL"] = nil
|
112
|
-
ENV["DATABASE_URL"] = "default_url"
|
113
|
-
end
|
114
|
-
|
115
|
-
it "defaults to using the DATABASE_URL" do
|
116
|
-
backup.send(:database_url).must_equal "default_url"
|
117
|
-
end
|
118
|
-
end
|
119
|
-
|
120
|
-
describe "an alternate database to backup is set" do
|
121
|
-
before do
|
122
|
-
ENV["PGBACKUPS_DATABASE_URL"] = "alternate_url"
|
123
|
-
ENV["DATABASE_URL"] = "default_url"
|
124
|
-
end
|
125
|
-
|
126
|
-
it "uses the PGBACKUPS_DATABASE_URL" do
|
127
|
-
backup.send(:database_url).must_equal "alternate_url"
|
128
|
-
end
|
129
|
-
end
|
130
|
-
end
|
131
|
-
|
132
|
-
describe "#file" do
|
133
|
-
let(:temp_file) { "temp-file" }
|
134
|
-
|
135
|
-
before do
|
136
|
-
backup.stubs(:temp_file).returns(temp_file)
|
137
|
-
File.expects(:open).with(temp_file, "r").returns("")
|
138
|
-
end
|
139
|
-
|
140
|
-
it { backup.send(:file) }
|
141
|
-
end
|
142
|
-
|
143
|
-
describe "#key" do
|
144
|
-
before do
|
145
|
-
backup.instance_eval do
|
146
|
-
@pgbackup = {
|
147
|
-
"finished_at" => "timestamp"
|
148
|
-
}
|
149
|
-
end
|
150
|
-
end
|
151
|
-
|
152
|
-
it "should be composed properly" do
|
153
|
-
path = ["pgbackups", backup.send(:environment), "timestamp.dump"]
|
154
|
-
.compact.join("/")
|
155
|
-
backup.send(:key).must_equal path
|
156
|
-
end
|
157
|
-
end
|
158
|
-
|
159
|
-
describe "#pgbackups_url" do
|
160
|
-
it { backup.send(:pgbackups_url).must_equal pgbackups_url }
|
161
|
-
end
|
162
|
-
|
163
|
-
describe "#temp_file" do
|
164
|
-
before do
|
165
|
-
backup.instance_eval do
|
166
|
-
@pgbackup = {
|
167
|
-
"public_url" => "https://raw.github.com/kjohnston/pgbackups-archive/master/pgbackups-archive.gemspec"
|
168
|
-
}
|
169
|
-
end
|
170
|
-
end
|
171
|
-
|
172
|
-
it "should be composed properly" do
|
173
|
-
temp_file = backup.send(:temp_file)
|
174
|
-
temp_file.must_match /^\/var\//
|
175
|
-
temp_file.must_match /\/pgbackups-archive.gemspec$/
|
176
|
-
end
|
177
|
-
end
|
178
|
-
|
179
|
-
end
|
180
|
-
|
181
|
-
end
|