braavos 0.0.7

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: f48996bb89970d302bac43a156684ad867b2fa13
4
+ data.tar.gz: 82b44ccb9e8ddc0fea2b01a0ad47a1f45a3d1ce6
5
+ SHA512:
6
+ metadata.gz: 04e073c3cce5c0c7dd0d0a25b26f9a114162e6bce51688f1ed2e20e95bfbdb6cb10c86082ffa43c1af2dd27c184a81250f76ce2d931c2ef0b0cfcb654ed961c8
7
+ data.tar.gz: 161846c8ee93cb51f3ea55f8b8088e945a636ebb16e51a7234b30dccf5bfacba429c7480608ad8d86daaee1068a58c75d79863306abe686d43030e5a1c148d10
data/Gemfile ADDED
@@ -0,0 +1,12 @@
1
+ source "http://rubygems.org"
2
+
3
+ gem 'gli'
4
+ gem 'aws-sdk'
5
+ gem 'rake'
6
+
7
+ group :test do
8
+ gem 'pry'
9
+ gem 'fakeweb'
10
+ gem 'simplecov', require: false
11
+ gem 'sqlite3'
12
+ end
data/Gemfile.lock ADDED
@@ -0,0 +1,44 @@
1
+ GEM
2
+ remote: http://rubygems.org/
3
+ specs:
4
+ aws-sdk (1.33.0)
5
+ json (~> 1.4)
6
+ nokogiri (>= 1.4.4)
7
+ uuidtools (~> 2.1)
8
+ coderay (1.1.0)
9
+ docile (1.1.0)
10
+ fakeweb (1.3.0)
11
+ gli (2.9.0)
12
+ json (1.8.1)
13
+ lockfile (2.1.0)
14
+ method_source (0.8.2)
15
+ mini_portile (0.5.3)
16
+ multi_json (1.8.4)
17
+ nokogiri (1.6.1)
18
+ mini_portile (~> 0.5.0)
19
+ pry (0.9.12.6)
20
+ coderay (~> 1.0)
21
+ method_source (~> 0.8)
22
+ slop (~> 3.4)
23
+ rake (10.1.1)
24
+ simplecov (0.8.1)
25
+ docile (~> 1.1.0)
26
+ lockfile (>= 2.1.0)
27
+ multi_json
28
+ simplecov-html (~> 0.8.0)
29
+ simplecov-html (0.8.0)
30
+ slop (3.5.0)
31
+ sqlite3 (1.3.8)
32
+ uuidtools (2.1.4)
33
+
34
+ PLATFORMS
35
+ ruby
36
+
37
+ DEPENDENCIES
38
+ aws-sdk
39
+ fakeweb
40
+ gli
41
+ pry
42
+ rake
43
+ simplecov
44
+ sqlite3
data/README.rdoc ADDED
@@ -0,0 +1,113 @@
1
+ = braavos - Backup/Restore Tool
2
+
3
+ == Introduction
4
+
5
+ This document covers the implementation of a backup/restore utility named braavos and deployed on Cassandra and ElasticSearch servers.
6
+ It implements:
7
+
8
+ * the management of backup data on S3
9
+ * Backup operations
10
+ * Restore operations
11
+ * Pruning executions
12
+ * full backup strategies (IMPLEMENTED)
13
+ * restore capabilities
14
+ * rewind to previous state
15
+ * restore fallen node (IMPLEMENTED)
16
+ * restore from different cluster (IMPLEMENTED)
17
+ * quick replacement strategy (using ebs volume)
18
+ * secondary backup strategy (google drive)
19
+
20
+ === Versions Targeted:
21
+ * Cassandra: 1.2
22
+ * ElasticSearch: 0.90
23
+
24
+ == Terminology
25
+
26
+ Service : Cassandra(cass) or ElasticSearch(es)
27
+
28
+ Node : A running instance of a service.
29
+
30
+ Cluster : A named cluster of nodes for a service.
31
+
32
+
33
+ == Design
34
+
35
+ Each service’s backups are maintained independently. They may or may not run on independent or shared hardware. Nodes are numbered and linked to their aws instance id.
36
+ A backup operation will validate the cluster name, environment and node name before proceeding with any mutation.
37
+
38
+ === Cluster Layout
39
+
40
+ cluster.json
41
+ {
42
+ "environment": "production",
43
+ "name": "place_directory",
44
+ "nodes": {
45
+ "1": {"instance_id": "i-abcdefed" },
46
+ "2": {"instance_id": "i-abcdefec" },
47
+ "3": {"instance_id": "i-abcdefeb" }
48
+ }
49
+ }
50
+
51
+
52
+ === Data Layout
53
+
54
+ General
55
+ /backup_bucket/name/environment/service/
56
+ /backup_bucket/name/environment/service/cluster.json
57
+ /backup_bucket/name/environment/service/full/datetime/node/
58
+ /backup_bucket/name/environment/service/full/datetime/node/_COMPLETED
59
+
60
+ Cassandra
61
+ /cassandra/data/node/keyspace/table/place_directory_production-Places-ic-218008.tgz ...
62
+ /cassandra/full/datetime/node/contents.json
63
+ /cassandra/full/datetime/node/system.tar.gz
64
+ /cassandra/full/datetime/node/schema.cql
65
+ /cassandra/full/datetime/node/ring.txt
66
+
67
+ ElasticSearch
68
+ /elasticsearch/full/datetime/node/cluster_state.local.json
69
+ /elasticsearch/full/datetime/node/cluster_state.global.json
70
+ /elasticsearch/full/datetime/node/index_name.restore.sh
71
+ /elasticsearch/full/datetime/node/index_name.tar.gz
72
+
73
+ ===Braavos Config
74
+
75
+ cassandra.yml
76
+ name: place_directory
77
+ environment: production
78
+ service: cassandra
79
+ bucket_name: backup_bucket
80
+ data_loc: /var/lib/cassandra/data
81
+ sync_loc: /mnt/cassandra_snapshots/latest
82
+
83
+ Operations
84
+
85
+ Initiate a full backup: (IMPLEMENTED)
86
+ braavos -c cassandra.yml backup full
87
+
88
+ List available backups: (IMPLEMENTED)
89
+ braavos -c cassandra.yml show
90
+ Verify backup:
91
+ braavos -c cassandra.yml verify backup_loc
92
+
93
+ Restore backup: (IMPLEMENTED)
94
+ braavos -c cassandra.yml restore backup_loc restore_loc
95
+
96
+ Restore across clusters: (IMPLEMENTED)
97
+ braavos -c cassandra.yml restore --cluster 'name:environment:node_id' backup_loc restore_loc
98
+ Example: ./bin/braavos -c config/braavos.yml restore --cluster 'place_directory:production:1' full/20140321 /tmp/restore
99
+
100
+ Prune backups:
101
+ braavos -c cassandra.yml prune
102
+
103
+ Sync latest (copy to ebs):
104
+ braavos sync -c cassandra.yml
105
+
106
+ == Mentions
107
+
108
+ braavos uses and depends on the following awesome components:
109
+
110
+ * {GNU Parallel}[http://www.gnu.org/software/parallel/] - The Command-Line Power Tool: O. Tange (2011)
111
+ * {GNU coreutils}[http://www.gnu.org/software/coreutils/] - The GNU Core Utils, including timeout.
112
+ * {s3cmd}[http://s3tools.org/s3cmd] - Command Line S3 Client. Please use version 1.5.0-beta1 or greater.
113
+ * {Ruby}[https://www.ruby-lang.org/] - A Programmer's Best Friend (including ERB templating)
data/Rakefile ADDED
@@ -0,0 +1,23 @@
1
+ require 'rake'
2
+ require 'rake/testtask'
3
+
4
+ require "bundler/gem_tasks"
5
+
6
+ Rake::TestTask.new(:test) do |t|
7
+ t.libs << 'test'
8
+ t.pattern = 'test/**/*_test.rb'
9
+ t.verbose = true
10
+ end
11
+
12
+ require 'rdoc/task'
13
+ desc 'Generate documentation.'
14
+ RDoc::Task.new(:rdoc) do |rdoc|
15
+ rdoc.rdoc_dir = 'rdoc'
16
+ rdoc.title = 'Braavos Backup/Restore Administrator'
17
+ rdoc.options << '--line-numbers' << '--inline-source'
18
+ rdoc.rdoc_files.include('README.rdoc')
19
+ rdoc.rdoc_files.include('lib/**/*.rb')
20
+ end
21
+
22
+ desc 'Default: run unit tests.'
23
+ task :default => :test
data/bin/braavos ADDED
@@ -0,0 +1,20 @@
1
+ #!/usr/bin/env ruby
2
+ # encoding: UTF-8
3
+
4
+ # resolve bin path, ignoring symlinks
5
+ require 'pathname'
6
+ bin_file = Pathname.new(__FILE__).realpath
7
+
8
+ # add locally vendored gems to libpath
9
+ gem_dir = File.expand_path('../../vendor/gems', bin_file)
10
+ Dir["#{gem_dir}/**/lib"].each do |libdir|
11
+ $:.unshift libdir
12
+ end
13
+
14
+ # add self to libpath
15
+ $:.unshift File.expand_path('../../lib', bin_file)
16
+
17
+ # start up the CLI
18
+ require 'braavos/cli'
19
+ exit Braavos::CLI.start(ARGV)
20
+
data/lib/braavos.rb ADDED
@@ -0,0 +1,179 @@
1
+ ENV["RACK_ENV"] ||= "development"
2
+ require 'bundler'
3
+
4
+ require 'rake'
5
+ require 'gli'
6
+ require 'aws-sdk'
7
+
8
+ require 'braavos/version'
9
+ require 'yaml'
10
+ require 'json'
11
+ require 'logger'
12
+ require 'set'
13
+
14
+ class Braavos
15
+
16
+ autoload :Command, 'braavos/command'
17
+ autoload :Config, 'braavos/config'
18
+ autoload :Parallel, 'braavos/parallel'
19
+ autoload :Service, 'braavos/service'
20
+ autoload :Storage, 'braavos/storage'
21
+ autoload :Template, 'braavos/template'
22
+
23
+ class << self
24
+ def config(config={})
25
+ @config ||= Config.new(config)
26
+ end
27
+
28
+ def command
29
+ @command ||= Command.new
30
+ end
31
+
32
+ def parallel
33
+ @parallel ||= Parallel.new("parallel", config.parallel_jobs || "100%")
34
+ end
35
+
36
+ def template
37
+ @template ||= Template.new([File.expand_path("../../template", __FILE__)])
38
+ end
39
+
40
+ def service
41
+ @service ||= config.service_class.new
42
+ end
43
+
44
+ def storage
45
+ @storage ||= config.storage_class.new
46
+ end
47
+
48
+ def logger
49
+ @logger ||= Logger.new(STDOUT)
50
+ end
51
+ end
52
+
53
+ attr_reader :global_options
54
+
55
+ def initialize(options)
56
+ # Load config
57
+ Braavos.logger.info "Braavos v#{Braavos::VERSION} Initializing"
58
+ Braavos.config(YAML.load(ERB.new(File.read(options[:config])).result))
59
+
60
+ initialize_aws(options)
61
+
62
+ Braavos.logger.level = Logger::INFO
63
+ if options[:verbose]
64
+ Braavos.logger.level = Logger::DEBUG
65
+ end
66
+ @global_options = options
67
+ Braavos.logger.debug "Options: #{options}"
68
+ Braavos.logger.debug "Config: #{Braavos.config.settings}"
69
+ AWS.config(:logger => Braavos.logger, :log_level => :debug)
70
+ end
71
+
72
+ def backup_full(args, options={})
73
+ Braavos.storage.verify_cluster(true)
74
+ if Braavos.service.respond_to? :backup_full
75
+ Braavos.service.backup_full
76
+ else
77
+ raise "Operation not supported by service #{Braavos.config.service}"
78
+ end
79
+ end
80
+
81
+ def restore(args, options={})
82
+ Braavos.storage.verify_cluster(true)
83
+
84
+ # restore location must be empty
85
+ # backup location must contain _COMPLETED
86
+
87
+ if args.length != 2
88
+ raise "Invalid restore args: backup_loc restore_loc"
89
+ end
90
+
91
+ restore_loc = args[1]
92
+ if File.exists?(restore_loc) && !File.directory?(restore_loc)
93
+ raise "restore_loc #{restore_loc} is not a directory"
94
+ end
95
+
96
+ FileUtils.mkdir_p restore_loc
97
+
98
+ if Dir["#{restore_loc}/*"].size > 1
99
+ raise "restore_loc #{restore_loc} directory must be empty"
100
+ end
101
+
102
+ if options[:cluster]
103
+ if match = options[:cluster].match(/(\w+):(\w+):(\d+)/)
104
+ name, environment, node = match.captures
105
+ Braavos.config.name = name
106
+ Braavos.config.environment = environment
107
+ Braavos.config.backup_path(true)
108
+ Braavos.storage.find_node_id(node)
109
+ else
110
+ raise "Invalid cluster config"
111
+ end
112
+ Braavos.logger.info("Restore across cluster: #{options[:cluster]}")
113
+ end
114
+
115
+ Braavos.service.restore(args[0], args[1])
116
+
117
+ end
118
+
119
+ def show(args, options={})
120
+ Braavos.logger.info "Data ks: #{Braavos.service.keyspaces_data}"
121
+ Braavos.logger.info "System ks: #{Braavos.service.keyspaces_system}"
122
+
123
+ Braavos.storage.verify_cluster(false)
124
+ node_id = ( options[:node_id] || Braavos.storage.find_node_id )
125
+ Braavos.logger.info "Node id: #{node_id}"
126
+ listing = Braavos.storage.list_backups(node_id)
127
+ puts "AVAILABLE BACKUPS:"
128
+ listing.each do |g, l|
129
+ puts
130
+ puts g
131
+ puts '-' * g.size
132
+ if l.size == 0
133
+ puts "NONE"
134
+ else
135
+ l.each do |name|
136
+ puts "\t#{name}"
137
+ end
138
+ end
139
+ end
140
+ end
141
+
142
+ def verify(args, options={})
143
+ end
144
+
145
+ def prune(args, options={})
146
+
147
+ end
148
+
149
+ def sync(args, options={})
150
+ Braavos.service.sync
151
+ end
152
+
153
+ def cleanup
154
+
155
+ end
156
+
157
+ def initialize_aws(options)
158
+ default_path = File.expand_path("../../config/aws.yml", __FILE__)
159
+
160
+ if options[:aws]
161
+ initialize_aws_from_file(options[:aws])
162
+ elsif File.exists?(default_path)
163
+ initialize_aws_from_file(default_path)
164
+ else
165
+ initialize_aws_from_environment
166
+ end
167
+ end
168
+
169
+ def initialize_aws_from_environment
170
+ Braavos.config.settings['aws'] = AWS.config.credentials
171
+ end
172
+
173
+ def initialize_aws_from_file(path)
174
+ raise ArgumentError, "AWS Credentials file not found" unless File.exists?(path)
175
+ aws_config = YAML.load(ERB.new(File.read(path)).result)
176
+ AWS.config(aws_config)
177
+ Braavos.config.settings['aws'] = aws_config
178
+ end
179
+ end
@@ -0,0 +1,82 @@
1
+ require 'gli'
2
+ require 'braavos'
3
+
4
+ class Braavos::CLI
5
+ extend GLI::App
6
+
7
+ class << self
8
+
9
+ def start(argv)
10
+ program_desc 'Backup/Restore Administrator'
11
+ sort_help :manually
12
+
13
+ version Braavos::VERSION
14
+
15
+ flag [:c,:config], desc: 'Configuration file', arg_name: 'config', required: true
16
+ flag [:a,:aws], desc: 'AWS Credentials file', arg_name: 'aws', required: false
17
+ switch [:v,:verbose], desc: 'Verbose mode'
18
+
19
+ pre do |global,command,options,args|
20
+ @braavos = Braavos.new(global)
21
+ true
22
+ end
23
+
24
+ desc 'Execute Backup'
25
+ command :backup do |c|
26
+ c.action do |global_options,options,args|
27
+ @braavos.backup_full(args, options)
28
+ end
29
+ end
30
+
31
+ desc 'Execute Restore'
32
+ command :restore do |c|
33
+ c.flag [:cluster], desc: 'Restore across cluster. Format: "cluster:environment:node"'
34
+
35
+ c.arg_name 'timestamp'
36
+ c.action do |global_options,options,args|
37
+ @braavos.restore(args, options)
38
+ end
39
+ end
40
+
41
+ desc 'Show Operation'
42
+ command :show do |c|
43
+ c.action do |global_options,options,args|
44
+ @braavos.show(args, options)
45
+ end
46
+ end
47
+
48
+ desc 'Verify Operation'
49
+ command :verify do |c|
50
+ c.action do |global_options,options,args|
51
+ @braavos.verify(args, options)
52
+ end
53
+ end
54
+
55
+ desc 'Sync Operation'
56
+ command :sync do |c|
57
+ c.action do |global_options,options,args|
58
+ @braavos.sync(args, options)
59
+ end
60
+ end
61
+
62
+ desc 'Prune Operation'
63
+ command :prune do |c|
64
+ c.action do |global_options,options,args|
65
+ @braavos.prune(args, options)
66
+ end
67
+ end
68
+
69
+ post do |global,command,options,args|
70
+ @braavos.cleanup
71
+ end
72
+
73
+ on_error do |exception|
74
+ @braavos.cleanup if @braavos
75
+ raise exception unless exception.is_a?(GLI::BadCommandLine)
76
+ true
77
+ end
78
+
79
+ run(argv)
80
+ end
81
+ end
82
+ end