easy_e 0.0.1

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: 37d1b662de1d5e139989991c7ddfe391a018e5a7
4
+ data.tar.gz: b73562bfc50c8fd642c4a310735b42cf81497c90
5
+ SHA512:
6
+ metadata.gz: 97a6e29177667f468bd52e56ecfecce3de4135f383cac8e90d4f3f4c4c8e18422767571b469edc0c533a766802c3ed0064e5ff8f024a304d64228ad8c8b693fd
7
+ data.tar.gz: 34b82f63039da516b16f9ae1a17aa3f5c2e7f828f6a3376937fc761a679b661a36667950df5e6f9bccc89352b23c15a674b2f25cd0d97fc3c844ab76ac16908f
data/README.md ADDED
@@ -0,0 +1,42 @@
1
+ easy-ec2-ebs-automatic-consistent-snapshot
2
+ ===
3
+
4
+ Easier to use than it is to say, this project aims to provide easy, automatic,
5
+ and consistent snapshots for AWS EBS volumes on EC2 instances.
6
+
7
+ Some specific goals and how they are achieved:
8
+
9
+ - *Safety*: refuses to operate unless everything seems ok, and tries desperately to leave your system no worse than it started
10
+ - *Reliability*: comprehensive test sweet makes sure that Easy-E behaves as expected, even in unexpected conditions.
11
+ - *Visibility*: verbose logging options to inspect the decision-making process for any action
12
+ - *Ease of Installation*: just install the gem and add one line to your crontab
13
+ - *Ease of Use*: automatically detects volumes mounted to the machine
14
+ - *Ease of Monitoring*: 100% visibility of operation can be gained from off-the-shelf monitoring solution plugins
15
+ - *Maintainability*: well-organized code structure and a modern language
16
+ - *Extensibility*: plugin architecture makes it easy to add lock support for services
17
+ - *Isolation*: plugin execution is isolated, so that an error in one is very unlikely to affect the others
18
+
19
+ Install
20
+ ===
21
+ ```
22
+ gem install easy-ec2-ebs-automatic-consistent-snapshot
23
+ crontab -e
24
+ ```
25
+
26
+ Testing
27
+ ===
28
+
29
+ Because you'll be running this against production servers with critical data, it's important that the functionality is well-tested. A thorough, pessimistic, multi-layer test suite hopes to assuage your concerns about letting a computer handle such an important task. The test suite is unquestionably the most complex part of this project.
30
+
31
+ Unit Tests
32
+ ---
33
+
34
+ Like any good Ruby software, this tool has a unit test suite that seeks mostly to verify the plumbing and ensure that there are no runtime errors on the expected execution paths. This is only the tip of the iceberg...
35
+
36
+ Vagrant Integration Testing
37
+ ---
38
+
39
+ The integration layer contains an Ansible + Vagrant setup to configure clusters of services for live-fire testing (the AWS bits are mocked out via Easy-E's `--mock` flag). Simply running `vagrant up` will build a cluster of servers running MySQL, MongoDB, etc, configured in a master/slave architecture as approprite for the given system.
40
+
41
+ There is also a set of Ansible tasks that verify the operation of each plugin under **both ideal and pathological** conditions. This means that Easy-E runs reliably, even when the services it operates on do not. Things like timeouts and service restart failures are modeled via `socat`, and assertions are made on the correct error output for each condition. For more info on how this is done, check `roles/integration-test/tasks/*.yml`
42
+
data/bin/easy-e ADDED
@@ -0,0 +1,4 @@
1
+ #!/usr/bin/env ruby
2
+ require './lib/easy_e.rb'
3
+
4
+ EasyE.new.execute
@@ -0,0 +1,46 @@
1
+ require 'optparse'
2
+
3
+ class EasyE
4
+ module Options
5
+ def option_parser
6
+ unless @option_parser
7
+ @option_parser = OptionParser.new do |o|
8
+ o.banner = "Usage: #{$0} [options]"
9
+
10
+ o.on("-v", "--[no-]verbose", "Run verbosely") do |val|
11
+ options[:verbose] = val
12
+ end
13
+
14
+ o.on("-a", "--access-key <AWS ACCESS KEY>", "AWS access key") do |val|
15
+ options[:access_key] = val
16
+ end
17
+
18
+ o.on("-s", "--secret-key <AWS SECRET KEY>", "AWS secret key") do |val|
19
+ options[:secret_key] = val
20
+ end
21
+
22
+ o.on("-c", "--credentials-file <FILE>", "Load AWS credentials from the downloaded CSV file (overrides -a and -s)") do |val|
23
+ options[:credentials_file] = val
24
+ end
25
+
26
+ o.on("-m", "--[no-]mock", "Mock out AWS calls for testing in Vagrant") do |val|
27
+ options[:mock] = val
28
+ end
29
+
30
+ o.on("-l", "--logfile FILE", "Path to a file used for logging") do |filename|
31
+ options.logfile = filename
32
+ logger.debug filename
33
+ end
34
+ end
35
+
36
+ plugins.each { |plugin| plugin.collect_options @option_parser }
37
+ end
38
+
39
+ @option_parser
40
+ end
41
+ end
42
+
43
+ def options
44
+ @options ||= OpenStruct.new
45
+ end
46
+ end
@@ -0,0 +1,49 @@
1
+ require 'ostruct'
2
+ class EasyE::Plugin
3
+ @@registered_plugins = []
4
+
5
+ attr_reader :options, :logger
6
+
7
+ def self.inherited(klass)
8
+ registered_plugins.unshift klass
9
+ end
10
+
11
+ def self.registered_plugins
12
+ @@registered_plugins
13
+ end
14
+
15
+ def default_options
16
+ { }
17
+ end
18
+
19
+ def options
20
+ @options ||= OpenStruct.new default_options
21
+ end
22
+
23
+ def logger
24
+ EasyE.logger false
25
+ end
26
+
27
+ def collect_options option_parser
28
+ option_parser.on "--#{name.downcase}", "Enable the #{name} plugin" do
29
+ options.enable = true
30
+ end
31
+
32
+ defined_options.each do |option_name, description|
33
+ option_parser.on "--#{name.downcase}-#{option_name.to_s.gsub('_','-')} #{option_name.upcase}", description do |val|
34
+ options[option_name.to_sym] = val
35
+ end
36
+ end
37
+ end
38
+
39
+ def carefully msg
40
+ yield
41
+ rescue Exception => e
42
+ logger.error "Error while trying to #{msg}"
43
+ logger.error e
44
+ nil
45
+ end
46
+ end
47
+
48
+ require 'plugins/mysql_plugin'
49
+ require 'plugins/mongo_plugin'
@@ -0,0 +1,48 @@
1
+ require 'csv'
2
+ require 'httparty'
3
+ require 'pp'
4
+ module EasyE::Snapshotter
5
+ AWS_INSTANCE_ID_URL = 'http://169.254.169.254/latest/dynamic/instance-identity/document'
6
+
7
+ attr_writer :storage, :compute, :instance_id
8
+ def take_snapshots
9
+ attached_volumes.collect do |vol|
10
+ logger.debug "Snapping #{vol.volume_id}"
11
+ snapshot = compute.snapshots.new
12
+ snapshot.volume_id = vol.volume_id
13
+ snapshot.save
14
+ snapshot
15
+ end
16
+ end
17
+
18
+ # lazy loaders
19
+ def compute
20
+ require 'fog/aws'
21
+ Fog.mock! if options[:mock]
22
+ @compute ||= Fog::Compute.new({
23
+ :aws_access_key_id => access_key,
24
+ :aws_secret_access_key => secret_key,
25
+ :provider => "AWS"
26
+ })
27
+ end
28
+
29
+ def attached_volumes
30
+ @attached_volumes ||= compute.volumes.select { |vol| vol.server_id == instance_id }
31
+ end
32
+
33
+ def access_key
34
+ @access_key ||= if options[:credentials_file] then credentials.first["Access Key Id"] else options[:access_key] end
35
+ end
36
+
37
+ def secret_key
38
+ @secret_key ||= if options[:credentials_file] then credentials.first["Secret Access Key"] else options[:secret_key] end
39
+ end
40
+
41
+ def credentials
42
+ @credentials ||= CSV.parse(File.read(options[:credentials_file]), :headers => true)
43
+ end
44
+
45
+ def instance_id
46
+ @instance_id ||= JSON.parse(HTTParty.get(AWS_INSTANCE_ID_URL))["instanceId"]
47
+ end
48
+ end
data/lib/easy_e.rb ADDED
@@ -0,0 +1,65 @@
1
+ $:.unshift File.dirname __FILE__
2
+ require 'logger'
3
+ require 'ostruct'
4
+ require 'easy_e/options'
5
+ require 'easy_e/snapshotter'
6
+ require 'easy_e/plugin'
7
+
8
+ class EasyE
9
+ include EasyE::Options
10
+ include EasyE::Snapshotter
11
+
12
+ @@logger = nil
13
+ def self.logger logfile
14
+ unless @@logger
15
+ @@logger = Logger.new(logfile || STDOUT)
16
+ @@logger.level = Logger::DEBUG
17
+ @@logger.formatter = proc do |severity, datetime, progname, msg|
18
+ "[#{severity}] #{datetime.strftime("%Y-%m-%d %H:%M:%S")} #{msg}\n"
19
+ end
20
+ end
21
+
22
+ @@logger
23
+ end
24
+
25
+ def plugins
26
+ @plugins ||= registered_plugins.collect { |klass| klass.new }
27
+ end
28
+
29
+ def registered_plugins
30
+ EasyE::Plugin.registered_plugins
31
+ end
32
+
33
+ def run
34
+ plugins.each do |plugin|
35
+ begin
36
+ plugin.before if plugin.options.enable
37
+ rescue Exception => e
38
+ logger.error "Encountered error while running the #{plugin.name} plugin's before hook"
39
+ logger.error e
40
+ end
41
+ end
42
+
43
+ take_snapshots
44
+
45
+ plugins.each do |plugin|
46
+ begin
47
+ plugin.after if plugin.options.enable
48
+ rescue Exception => e
49
+ logger.error "Encountered error while running the #{plugin.name} plugin's after hook"
50
+ logger.error e
51
+ end
52
+ end
53
+ end
54
+
55
+ def execute
56
+ option_parser.parse!
57
+ logger.debug "Debug logging enabled"
58
+ run
59
+ end
60
+
61
+ def logger
62
+ # HACK -- the logfile argument only gets used on the first invocation
63
+ EasyE.logger options.logfile
64
+ end
65
+ end
@@ -0,0 +1,125 @@
1
+ require 'pp'
2
+ class EasyE::Plugin::MongoPlugin < EasyE::Plugin
3
+ WIRED_TIGER_KEY = 'wiredTiger'
4
+ attr_accessor :client
5
+ def defined_options
6
+ {
7
+ service: 'Service to start after shutting down server',
8
+ shutdown: 'Shutdown mongodb server (this is required if your data and journal are on different volumes',
9
+ user: 'Mongo user',
10
+ password: 'Mongo password',
11
+ port: 'Mongo port',
12
+ host: 'Mongo host',
13
+ server_selection_timeout: 'Timeout in seconds while choosing a server to connect to (default 30)',
14
+ wait_queue_timeout: 'Timeout in seconds while waiting for a connection in the pool (default 1)',
15
+ connection_timeout: 'Timeout in seconds to wait for a socket to connect (default 5)',
16
+ socket_timeout: 'Timeout in seconds to wait for an operation to execute on a socket (default 5)'
17
+ }
18
+ end
19
+
20
+ def default_options
21
+ {
22
+ service: 'mongodb',
23
+ port: '27017',
24
+ shutdown: false,
25
+ host: 'localhost',
26
+ server_selection_timeout: 30,
27
+ wait_queue_timeout: 1,
28
+ connection_timeout: 5,
29
+ socket_timeout: 5
30
+ }
31
+ end
32
+
33
+ def client
34
+ @client ||= Mongo::Client.new [ "#{options.host}:#{options.port}" ], client_options
35
+ end
36
+
37
+ def client_options
38
+ {
39
+ user: options.user,
40
+ password: options.password,
41
+ server_selection_timeout: options.server_selection_timeout.to_i,
42
+ wait_queue_timeout: options.wait_queue_timeout.to_i,
43
+ connection_timeout: options.connection_timeout.to_i,
44
+ socket_timeout: options.socket_timeout.to_i
45
+ }
46
+ end
47
+
48
+ def before
49
+ require 'mongo'
50
+ Mongo::Logger.logger = logger
51
+ return logger.error "Refusing to operate" if carefully('check whether this node is a primary') { primary? }.nil?
52
+ return logger.error "This appears to be a primary member, refusing to operate" if primary?
53
+
54
+ if wired_tiger?
55
+ logger.info "Wired Tiger storage engine detected"
56
+ carefully('shutdown mongo') { shutdown_mongo } if options.shutdown
57
+ else
58
+ logger.info "MMAPv1 storage engine detected"
59
+ carefully('lock mongo') { lock_mongo }
60
+ end
61
+ end
62
+
63
+ def after
64
+ return logger.error "Refusing to operate" if carefully('check whether this node is a primary') { primary? }.nil?
65
+ return logger.error "This appears to be a primary member, refusing to operate" if primary?
66
+
67
+ if wired_tiger?
68
+ carefully('start mongo') { start_mongo } if options.shutdown
69
+ else
70
+ carefully('unlock mongo') { unlock_mongo }
71
+ end
72
+
73
+ if carefully('check that mongo is still accessible') { client.command(serverStatus: 1).first }
74
+ logger.info "Received status from mongo, everything appears to be ok"
75
+ end
76
+ end
77
+
78
+ def name
79
+ "Mongo"
80
+ end
81
+
82
+ private
83
+
84
+ def wired_tiger?
85
+ if @wired_tiger.nil?
86
+ @wired_tiger = client.command(serverStatus: 1).first.has_key? WIRED_TIGER_KEY
87
+ end
88
+ @wired_tiger
89
+ end
90
+
91
+ def primary?
92
+ if @primary.nil?
93
+ @primary = client.command(isMaster: 1).first['ismaster']
94
+ end
95
+ @primary
96
+ end
97
+
98
+ def shutdown_mongo
99
+ logger.info 'Shutting down mongodb'
100
+ begin
101
+ # this will always raise an exception after it completes
102
+ client.command shutdown: 1
103
+ rescue Mongo::Error::SocketError => e
104
+ logger.debug "Received expected socket error after shutting down"
105
+ end
106
+
107
+ # we need a new connection now since the server has shut down
108
+ @client = nil
109
+ end
110
+
111
+ def start_mongo
112
+ logger.info "Starting mongodb via 'service #{options[:service]} start'"
113
+ system "service #{options[:service]} start"
114
+ end
115
+
116
+ def lock_mongo
117
+ logger.info "Locking mongo"
118
+ client.command(fsync: 1, lock: true)
119
+ end
120
+
121
+ def unlock_mongo
122
+ logger.info "Unlocking mongo"
123
+ client.database['$cmd.sys.unlock'].find().first
124
+ end
125
+ end
@@ -0,0 +1,22 @@
1
+ class EasyE::Plugin::MysqlPlugin < EasyE::Plugin
2
+ def defined_options
3
+ {
4
+ user: 'MySql Username',
5
+ pass: 'MySql Password',
6
+ port: 'MySql port',
7
+ host: 'MySql host'
8
+ }
9
+ end
10
+
11
+ def before
12
+ require 'mysql'
13
+ Mysql.new
14
+ end
15
+
16
+ def after
17
+ end
18
+
19
+ def name
20
+ "Mysql"
21
+ end
22
+ end
metadata ADDED
@@ -0,0 +1,52 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: easy_e
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.0.1
5
+ platform: ruby
6
+ authors:
7
+ - Bryan Conrad
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2015-04-21 00:00:00.000000000 Z
12
+ dependencies: []
13
+ description: Easy EBS snapshots that work
14
+ email: bryan.conrad@synctree.com
15
+ executables:
16
+ - easy-e
17
+ extensions: []
18
+ extra_rdoc_files: []
19
+ files:
20
+ - README.md
21
+ - bin/easy-e
22
+ - lib/easy_e.rb
23
+ - lib/easy_e/options.rb
24
+ - lib/easy_e/plugin.rb
25
+ - lib/easy_e/snapshotter.rb
26
+ - lib/plugins/mongo_plugin.rb
27
+ - lib/plugins/mysql_plugin.rb
28
+ homepage: http://rubygems.org/gems/easy_e
29
+ licenses:
30
+ - MIT
31
+ metadata: {}
32
+ post_install_message:
33
+ rdoc_options: []
34
+ require_paths:
35
+ - lib
36
+ required_ruby_version: !ruby/object:Gem::Requirement
37
+ requirements:
38
+ - - ">="
39
+ - !ruby/object:Gem::Version
40
+ version: '0'
41
+ required_rubygems_version: !ruby/object:Gem::Requirement
42
+ requirements:
43
+ - - ">="
44
+ - !ruby/object:Gem::Version
45
+ version: '0'
46
+ requirements: []
47
+ rubyforge_project:
48
+ rubygems_version: 2.4.6
49
+ signing_key:
50
+ specification_version: 4
51
+ summary: Easy EBS snapshots that work
52
+ test_files: []