markmansour-safe 0.1.7

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
data/LICENSE ADDED
@@ -0,0 +1,20 @@
1
+ Copyright (c) 2009 Astrails Ltd.
2
+
3
+ Permission is hereby granted, free of charge, to any person obtaining
4
+ a copy of this software and associated documentation files (the
5
+ "Software"), to deal in the Software without restriction, including
6
+ without limitation the rights to use, copy, modify, merge, publish,
7
+ distribute, sublicense, and/or sell copies of the Software, and to
8
+ permit persons to whom the Software is furnished to do so, subject to
9
+ the following conditions:
10
+
11
+ The above copyright notice and this permission notice shall be
12
+ included in all copies or substantial portions of the Software.
13
+
14
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
15
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
16
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
17
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
18
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
19
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
20
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,171 @@
1
+ astrails-safe
2
+ =============
3
+
4
+ Simple database and filesystem backups with S3 support (with optional encryption)
5
+
6
+ Motivation
7
+ ----------
8
+
9
+ We needed a backup solution that will satisfy the following requirements:
10
+
11
+ * opensource
12
+ * simple to install and configure
13
+ * support for simple ‘tar’ backups of directories (with includes/excludes)
14
+ * support for simple mysqldump of mysql databases
15
+ * support for simple pg_dump of PostgreSQL databases
16
+ * support for symmetric or public key encryption
17
+ * support for local filesystem and Amazon S3 for storage
18
+ * support for backup rotation. we don’t want backups filling all the diskspace or cost a fortune on S3
19
+
20
+ And since we didn't find any, we wrote our own :)
21
+
22
+ Usage
23
+ -----
24
+
25
+ Usage:
26
+ astrails-safe [OPTIONS] CONFIG_FILE
27
+ Options:
28
+ -h, --help This help screen
29
+ -v, --verbose be verbose, duh!
30
+ -n, --dry-run just pretend, don't do anything.
31
+ -L, --local skip S3
32
+
33
+ Note: CONFIG_FILE will be created from template if missing
34
+
35
+ Encryption
36
+ ----------
37
+
38
+ If you want to encrypt your backups you have 2 options:
39
+ * use simple password encryption
40
+ * use GPG public key encryption
41
+
42
+ For simple password, just add password entry in gpg section.
43
+ For public key encryption you will need to create a public/secret keypair.
44
+
45
+ We recommend to create your GPG keys only on your local machine and then
46
+ transfer your public key to the server that will do the backups.
47
+
48
+ This way the server will only know how to encrypt the backups but only you
49
+ will be able to decrypt them using the secret key you have locally. Of course
50
+ you MUST backup your backup encryption key :)
51
+ We recommend also pringing the hard paper copy of your GPG key 'just in case'.
52
+
53
+ The procedure to create and transfer the key is as follows:
54
+
55
+ 1. run 'gpg --gen-gen' on your local machine and follow onscreen instructions to create the key
56
+ (you can accept all the defaults).
57
+
58
+ 2. extract your public key into a file (assuming you used test@example.com as your key email):
59
+ gpg -a --export test@example.com > test@example.com.pub
60
+
61
+ 3. transfer public key to the server
62
+ scp backup@example.com root@example.com:
63
+
64
+ 4. import public key on the remote system:
65
+ <pre>
66
+ $ gpg --import test@example.com.pub
67
+ gpg: key 45CA9403: public key "Test Backup <test@example.com>" imported
68
+ gpg: Total number processed: 1
69
+ gpg: imported: 1
70
+ </pre>
71
+
72
+ 5. since we don't keep the secret part of the key on the remote server, gpg has
73
+ no way to know its yours and can be trusted.
74
+ To fix that we can sign it with other trusted key, or just directly modify its
75
+ trust level in gpg (use level 5):
76
+ <pre>
77
+ $ gpg --edit-key test@example.com
78
+ ...
79
+ Command> trust
80
+ ...
81
+ 1 = I don't know or won't say
82
+ 2 = I do NOT trust
83
+ 3 = I trust marginally
84
+ 4 = I trust fully
85
+ 5 = I trust ultimately
86
+ m = back to the main menu
87
+
88
+ Your decision? 5
89
+ ...
90
+ Command> quit
91
+ </pre>
92
+
93
+ 6. export your secret key for backup
94
+ (we recommend to print it on paper and burn to a CD/DVD and store in a safe place):
95
+ <pre>
96
+ $ gpg -a --export-secret-key test@example.com > test@example.com.key
97
+ </pre>
98
+
99
+
100
+ Example configuration
101
+ ---------------------
102
+ <pre>
103
+ safe do
104
+ local :path => "/backup/:kind/:id"
105
+
106
+ s3 do
107
+ key "...................."
108
+ secret "........................................"
109
+ bucket "backup.astrails.com"
110
+ path "servers/alpha/:kind/:id"
111
+ end
112
+
113
+ gpg do
114
+ # symmetric encryption key
115
+ # password "qwe"
116
+
117
+ # public GPG key (must be known to GPG, i.e. be on the keyring)
118
+ key "backup@astrails.com"
119
+ end
120
+
121
+ keep do
122
+ local 2
123
+ s3 2
124
+ end
125
+
126
+ mysqldump do
127
+ options "-ceKq --single-transaction --create-options"
128
+
129
+ user "root"
130
+ password "............"
131
+ socket "/var/run/mysqld/mysqld.sock"
132
+
133
+ database :blog
134
+ database :servershape
135
+ database :astrails_com
136
+ database :secret_project_com
137
+
138
+ end
139
+
140
+ pgdump do
141
+ options "-i -x -O" # -i => ignore version, -x => do not dump privileges (grant/revoke), -O => skip restoration of object ownership in plain text format
142
+
143
+ user "username"
144
+ password "............" # shouldn't be used, instead setup ident. Current functionality exports a password env to the shell which pg_dump uses - untested!
145
+
146
+ database :blog
147
+ database :stateofflux_com
148
+ end
149
+
150
+ tar do
151
+ archive "git-repositories", :files => "/home/git/repositories"
152
+ archive "dot-configs", :files => "/home/*/.[^.]*"
153
+ archive "etc", :files => "/etc", :exclude => "/etc/puppet/other"
154
+
155
+ archive "blog-astrails-com" do
156
+ files "/var/www/blog.astrails.com/"
157
+ exclude ["/var/www/blog.astrails.com/log", "/var/www/blog.astrails.com/tmp"]
158
+ end
159
+
160
+ archive "astrails-com" do
161
+ files "/var/www/astrails.com/"
162
+ exclude ["/var/www/astrails.com/log", "/var/www/astrails.com/tmp"]
163
+ end
164
+ end
165
+ end
166
+ </pre>
167
+
168
+ Copyright
169
+ ---------
170
+
171
+ Copyright (c) 2009 Astrails Ltd. See LICENSE for details.
@@ -0,0 +1,52 @@
1
+ require 'rubygems'
2
+ require 'rake'
3
+
4
+ begin
5
+ require 'jeweler'
6
+ Jeweler::Tasks.new do |gem|
7
+ gem.name = "safe"
8
+ gem.summary = %Q{Backup filesystem and databases (MySQL and PostgreSQL) to Amazon S3 (with encryption)}
9
+ gem.description = "Simple tool to backup databases (MySQL and PostgreSQL) and filesystem locally or to Amazon S3 (with optional encryption)"
10
+ gem.email = "we@astrails.com"
11
+ gem.homepage = "http://github.com/astrails/safe"
12
+ gem.authors = ["Astrails Ltd.", "Mark Mansour"]
13
+ gem.files = FileList["[A-Z]*.*", "{bin,examples,generators,lib,rails,spec,test,templates}/**/*", 'Rakefile', 'LICENSE*']
14
+
15
+ gem.add_dependency("aws-s3")
16
+
17
+ # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
18
+ end
19
+ rescue LoadError
20
+ puts "Jeweler not available. Install it with: sudo gem install technicalpickles-jeweler -s http://gems.github.com"
21
+ end
22
+
23
+ require 'micronaut/rake_task'
24
+ Micronaut::RakeTask.new(:examples) do |examples|
25
+ examples.pattern = 'examples/**/*_example.rb'
26
+ examples.ruby_opts << '-Ilib -Iexamples'
27
+ end
28
+
29
+ Micronaut::RakeTask.new(:rcov) do |examples|
30
+ examples.pattern = 'examples/**/*_example.rb'
31
+ examples.rcov_opts = '-Ilib -Iexamples'
32
+ examples.rcov = true
33
+ end
34
+
35
+
36
+ task :default => :examples
37
+
38
+ require 'rake/rdoctask'
39
+ Rake::RDocTask.new do |rdoc|
40
+ if File.exist?('VERSION.yml')
41
+ config = YAML.load(File.read('VERSION.yml'))
42
+ version = "#{config[:major]}.#{config[:minor]}.#{config[:patch]}"
43
+ else
44
+ version = ""
45
+ end
46
+
47
+ rdoc.rdoc_dir = 'rdoc'
48
+ rdoc.title = "safe #{version}"
49
+ rdoc.rdoc_files.include('README*')
50
+ rdoc.rdoc_files.include('lib/**/*.rb')
51
+ end
52
+
@@ -0,0 +1,4 @@
1
+ ---
2
+ :major: 0
3
+ :minor: 1
4
+ :patch: 7
@@ -0,0 +1,59 @@
1
+ #!/usr/bin/env ruby
2
+
3
+
4
+ require 'tempfile'
5
+ require 'rubygems'
6
+ require 'fileutils'
7
+ require "aws/s3"
8
+ require 'yaml'
9
+
10
+ #require 'ruby-debug'
11
+ #$:.unshift File.join(File.dirname(__FILE__), "..", "lib")
12
+
13
+ require 'astrails/safe'
14
+ include Astrails::Safe
15
+
16
+ def die(msg)
17
+ puts "ERROR: #{msg}"
18
+ exit 1
19
+ end
20
+
21
+ def usage
22
+ puts <<-END
23
+ Usage: astrails-safe [OPTIONS] CONFIG_FILE
24
+ Options:
25
+ -h, --help This help screen
26
+ -v, --verbose be verbose, duh!
27
+ -n, --dry-run just pretend, don't do anything.
28
+ -L, --local skip S3
29
+
30
+ Note: config file will be created from template if missing
31
+ END
32
+ exit 1
33
+ end
34
+
35
+ def process_options
36
+ usage if ARGV.delete("-h") || ARGV.delete("--help")
37
+ $_VERBOSE = ARGV.delete("-v") || ARGV.delete("--verbose")
38
+ $DRY_RUN = ARGV.delete("-n") || ARGV.delete("--dry-run")
39
+ $LOCAL = ARGV.delete("-L") || ARGV.delete("--local")
40
+ usage unless ARGV.first
41
+ $CONFIG_FILE_NAME = File.expand_path(ARGV.first)
42
+ end
43
+
44
+ def main
45
+ process_options
46
+
47
+ unless File.exists?($CONFIG_FILE_NAME)
48
+ die "Missing configuration file. NOT CREATED! Rerun w/o the -n argument to create a template configuration file." if $DRY_RUN
49
+
50
+ FileUtils.cp File.join(Astrails::Safe::ROOT, "templates", "script.rb"), $CONFIG_FILE_NAME
51
+
52
+ die "Created default #{$CONFIG_FILE_NAME}. Please edit and run again."
53
+ end
54
+
55
+
56
+ load($CONFIG_FILE_NAME)
57
+ end
58
+
59
+ main
@@ -0,0 +1,19 @@
1
+ require 'rubygems'
2
+ require 'micronaut'
3
+ require 'ruby-debug'
4
+
5
+ $LOAD_PATH.unshift(File.dirname(__FILE__))
6
+ $LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
7
+
8
+ require 'astrails/safe'
9
+
10
+ def not_in_editor?
11
+ !(ENV.has_key?('TM_MODE') || ENV.has_key?('EMACS') || ENV.has_key?('VIM'))
12
+ end
13
+
14
+ Micronaut.configure do |c|
15
+ c.color_enabled = not_in_editor?
16
+ c.filter_run :focused => true
17
+ c.mock_with :rr
18
+ end
19
+
@@ -0,0 +1,175 @@
1
+ require File.expand_path(File.dirname(__FILE__) + '/../example_helper')
2
+
3
+ describe Astrails::Safe::Config do
4
+ it "pgdump" do
5
+ config = Astrails::Safe::Config::Node.new do
6
+ local do
7
+ path "path"
8
+ end
9
+
10
+ pgdump do
11
+ # `pg_dump -U "#{abcs[RAILS_ENV]["username"]}" -i -x -O #{abcs[RAILS_ENV]["database"]} -f db/#{filename}`
12
+ options "-i -x -O"
13
+
14
+ user "astrails"
15
+ password ""
16
+ host "localhost"
17
+ port 5432
18
+
19
+ database :blog
20
+
21
+ database :production do
22
+ keep :local => 3
23
+
24
+ skip_tables [:logger_exceptions, :request_logs]
25
+ end
26
+
27
+ end
28
+ end
29
+
30
+ expected = {
31
+ "local" => {"path" => "path"},
32
+
33
+ "pgdump" => {
34
+ "options" => "-i -x -O",
35
+ "user" => "astrails",
36
+ "password" => "",
37
+ "host" => "localhost",
38
+ "port" => 5432,
39
+
40
+ "databases" => {
41
+ "blog" => {},
42
+ "production" => {
43
+ "keep" => {"local" => 3},
44
+ "skip_tables" => ["logger_exceptions", "request_logs"],
45
+ },
46
+ },
47
+ }
48
+ }
49
+
50
+ config.to_hash.should == expected
51
+ end
52
+
53
+ it "foo" do
54
+ config = Astrails::Safe::Config::Node.new do
55
+ local do
56
+ path "path"
57
+ end
58
+
59
+ s3 do
60
+ key "s3 key"
61
+ secret "secret"
62
+ bucket "bucket"
63
+ path "path1"
64
+ end
65
+
66
+ gpg do
67
+ key "gpg-key"
68
+ password "astrails"
69
+ end
70
+
71
+ keep do
72
+ local 4
73
+ s3 20
74
+ end
75
+
76
+ mysqldump do
77
+ options "-ceKq --single-transaction --create-options"
78
+
79
+ user "astrails"
80
+ password ""
81
+ host "localhost"
82
+ port 3306
83
+ socket "/var/run/mysqld/mysqld.sock"
84
+
85
+ database :blog
86
+
87
+ database :production do
88
+ keep :local => 3
89
+
90
+ gpg do
91
+ password "custom-production-pass"
92
+ end
93
+
94
+ skip_tables [:logger_exceptions, :request_logs]
95
+ end
96
+
97
+ end
98
+
99
+
100
+ tar do
101
+ archive "git-repositories" do
102
+ files "/home/git/repositories"
103
+ end
104
+
105
+ archive "etc-files" do
106
+ files "/etc"
107
+ exclude "/etc/puppet/other"
108
+ end
109
+
110
+ archive "dot-configs" do
111
+ files "/home/*/.[^.]*"
112
+ end
113
+
114
+ archive "blog" do
115
+ files "/var/www/blog.astrails.com/"
116
+ exclude ["/var/www/blog.astrails.com/log", "/var/www/blog.astrails.com/tmp"]
117
+ end
118
+
119
+ archive :misc do
120
+ files [ "/backup/*.rb" ]
121
+ end
122
+ end
123
+
124
+ end
125
+
126
+ expected = {
127
+ "local" => {"path" => "path"},
128
+
129
+ "s3" => {
130
+ "key" => "s3 key",
131
+ "secret" => "secret",
132
+ "bucket" => "bucket",
133
+ "path" => "path1",
134
+ },
135
+
136
+ "gpg" => {"password" => "astrails", "key" => "gpg-key"},
137
+
138
+ "keep" => {"s3" => 20, "local" => 4},
139
+
140
+ "mysqldump" => {
141
+ "options" => "-ceKq --single-transaction --create-options",
142
+ "user" => "astrails",
143
+ "password" => "",
144
+ "host" => "localhost",
145
+ "port" => 3306,
146
+ "socket" => "/var/run/mysqld/mysqld.sock",
147
+
148
+ "databases" => {
149
+ "blog" => {},
150
+ "production" => {
151
+ "keep" => {"local" => 3},
152
+ "gpg" => {"password" => "custom-production-pass"},
153
+ "skip_tables" => ["logger_exceptions", "request_logs"],
154
+ },
155
+ },
156
+ },
157
+ "tar" => {
158
+ "archives" => {
159
+ "git-repositories" => {"files" => "/home/git/repositories"},
160
+ "etc-files" => {"files" => "/etc", "exclude" => "/etc/puppet/other"},
161
+ "dot-configs" => {"files" => "/home/*/.[^.]*"},
162
+ "blog" => {
163
+ "files" => "/var/www/blog.astrails.com/",
164
+ "exclude" => ["/var/www/blog.astrails.com/log", "/var/www/blog.astrails.com/tmp"],
165
+ },
166
+ "misc" => { "files" => ["/backup/*.rb"] },
167
+ },
168
+ },
169
+ }
170
+
171
+ config.to_hash.should == expected
172
+
173
+ end
174
+ end
175
+