mongo-ec2-backup 0.0.4 → 0.0.5

data/README.markdown

@@ -2,67 +2,38 @@
 
 Suite of tools to backup and manage snapshots of MongoDB data set to EC2 Snapshots.
 
-## Lock and Snapshot: lock_and_snapshot.rb
+## Lock and Snapshot: mongo_lock_and_snapshot.rb
 
 ### Usage
 
-Snapshot a list of devices on a given instance on ec2. Requires network access in order to lock and unlock Mongo
+Snapshot a list of devices on a given instance on ec2.
 
 ```shell
-./lock_and_snapshot.rb -a ACCESS_KEY_ID -s SECRET_ACCESS_KEY --hostname server01 --devices /dev/sdl,/dev/slm --type daily --limit 4
+/mnt/lib/mongo-ec2-consistent-backup/bin# infra-ruby lock_and_snapshot -p /ebs/lvms/lvol0/ -a ACCESS_KEY -s SECRET_KEY -d /dev/sdg,/dev/sdh,/dev/sdi,/dev/sdj,/dev/sdk,/dev/sdl,/dev/sdm,/dev/sdn
 ```
 
-* --port, -p <i>:   Mongo port to connect to (default: 27017)
-* --access-key-id, -a <s>:   Access Key Id for AWS
-* --secret-access-key, -s <s>:   Secret Access Key for AWS
-* --devices, -d <s>:   Devices to snapshot, comma separated
-* --hostname, -h <s>:   Hostname to look for. Should resolve to a local EC2 Ip
-* --type, -t <s>:   Snapshot type, to choose among snapshot,weekly,monthly,daily,yearly (default: snapshot)
-* --limit, -l <i>:   Cleanup old snapshots to keep only limit snapshots. Default values are stored in EC2VolumeSnapshoter::KIND
-* --region:   Region hosting the instances
-* --help, -e:   Show this message
+* --path, -p :   Data path to freeze
+* --access-key-id, -a :   Access Key Id for AWS
+* --secret-access-key, -s :   Secret Access Key for AWS
+* --devices, -d :   Devices to snapshot, comma separated
+* --type, -t :   Snapshot type, to choose among test,snapshot,daily,weekly,monthly,yearly (default: snapshot)
+* --help, -h:   Show this message
 
-### Usage in chef environment
+It freeze the path using ```xfs_freeze```and create a snapshot for all the disks passed in the command line. To make this work without too much trouble with a mongo that is in a replica set, you can shut down the replica before running the command. You can also use mongo fsync and lock but this will probably make your cluster a bit nervous about that. Shutting down ensure no mongos will try to use the frozen mongo.
 
-In order to run the command from a remote server (the Chef server or any administrative node of your grid), you need to be able to know the lists of the devices you wish to snapshot.
+### Usage with IAM
 
-By using the ohai-raid plugin (https://github.com/octplane/ohai-raid), Chef clients can fill part of their Chef registry with information about the software managed RAID arrays running.
-This information can be fetched out for use at a later point via the knife script provided in the ohai-raid package:
+If you use IAM for your authentication in EC2, here is a probably up to date list of the permissions you need to grant:
 
 ```
-knife exec scripts/show_raid_devices server01.fqdn.com /dev/md0
-/dev/sdl,/dev/sdm,/dev/sdn,/dev/sdo
+  "ec2:CreateSnapshot",
+  "ec2:DeleteSnapshot",
+  "ec2:DescribeSnapshots",
+  "ec2:CreateTags",
+  "ec2:DescribeTags",
+  "ec2:DescribeVolumes",
+  "ec2:DescribeInstances",
+  "ec2:AttachVolume",
+  "ec2:CreateVolume"
 ```
-
-You can combine the two tools to automate daily backup of you MongoDB server:
-
-```
-./lock_and_snapshot.rb -a ACCESS_KEY_ID -s SECRET_ACCESS_KEY --hostname server01 --devices $(knife exec /path/to/scripts/show_raid_devices server01.fqdn.com /dev/md0) --type daily
-```
-
-### Tool Description
-
-* Find instance id by resolving the hostname provided in the CLI and scanning the instances in EC2
-* Lock Mongo by connecting via the hostname:port provided in the parameters
-* Snapshot the disks, delete old backups
-* Unlock Mongo
-
-## MD inspection: ec2-consistent-backup.rb
-
-### Usage
-
-This script demonstrates the way it analyses Mongo DB Data path to extract the MD device and components associated
-
-```shell
-./ec2-consistent-backup -p 27017 
-```
-
-### Tool description
-
-* connect to mongo at port provided, retrieves dbpath
-* find what mount this dbpath corresponds to
-* use /proc/mdstat to find out which drive are corresponding to the dbpath mount disk
-
-# API
-
 Internal API documentation is at: http://rubydoc.info/github/octplane/mongo-ec2-consistent-backup/master/frames

data/Rakefile

@@ -3,12 +3,12 @@ require "rake/clean"
 require 'rake/gempackagetask'
 require 'rake/rdoctask'
 
-desc "Packages up Swissr."
+desc "Packages up the gem."
 task :default => :package
 
 spec = Gem::Specification.new do |s|
   s.name    = 'mongo-ec2-backup'
-  s.version = '0.0.4'
+  s.version = '0.0.5'
   s.summary = 'Snapshot your mongodb in the EC2 cloud via XFS Freeze'
 
   s.author   = 'Pierre Baillet'
@@ -25,6 +25,7 @@ spec = Gem::Specification.new do |s|
   s.files = FileList['lib/**/*.rb', 'bin/*', '[A-Z]*', 'test/**/*'].to_a
 
   s.executables << "mongo_lock_and_snapshot"
+  s.executables << "ec2_snapshot_restorer"
 
   # Supress the warning about no rubyforge project
   s.rubyforge_project = 'nowarning'

data/bin/ec2_snapshot_restorer

@@ -0,0 +1,132 @@
+#!/usr/bin/env ruby
+
+require 'rubygems'
+ENV['BUNDLE_GEMFILE'] =  File.join(File.dirname(__FILE__), "..", "Gemfile")
+require 'bundler/setup'
+require 'trollop'
+require 'fog'
+require 'open-uri'
+
+$: << File.join(File.dirname(__FILE__), "../lib")
+
+DEBUG=false
+
+def log what
+  puts what if DEBUG
+end
+# 
+class SnapshotRestorer
+  attr_accessor :snaps
+  def initialize(aki, sak)
+    @compute = Fog::Compute.new({:provider => 'AWS', :aws_access_key_id => aki, :aws_secret_access_key => sak})
+    @snaps = []
+    @volumes = []
+  end
+  def find_snapshots(instance_id, kind = 'snapshot')
+    log "Looking for snapshots for #{instance_id}"
+    volume_map = []
+    snapshots = {}
+
+    tags = @compute.tags.all(:key => 'instance_id', :value => instance_id)
+
+    max_date = nil
+    tags.each do |tag|
+      snap = @compute.snapshots.get(tag.resource_id)
+      t =  snap.tags
+
+      # Ignore in progress snapshots
+      if instance_id == t['instance_id'] && 
+          snap.state == 'completed' &&
+          t['kind'] == kind
+        max_date = t['date'] if !max_date || max_date < t['date']
+        log "#{snap.inspect} is valid"
+        snapshots[t['date']] ||= []
+        snapshots[t['date']] << snap
+      end
+    end
+    snapshots['LATEST'] = snapshots[max_date] if snapshots[max_date]
+    return snapshots
+  end
+  def prepare_volumes(dest_instance)
+    @snaps.each do | resource_id |
+      snap = @compute.snapshots.get(resource_id)
+      # Snap have the following tags
+      # application
+      # device
+      # instance_id
+      # date
+      # kind
+
+      t =  snap.tags
+      volume = @compute.volumes.new :snapshot_id => snap.id, :size => snap.volume_size, :availability_zone => 'us-east-1c'
+      volume.save
+      volume.reload
+      @compute.create_tags(volume.id, { "application" => t['application'],
+        "sdevice" => t['device'],
+        "date" => t['date'],
+        "kind" => t['kind'],
+        "sinstance" => t['instance_id'],
+        "dinstance" =>  dest_instance})
+      
+      @volumes << volume
+    end
+    def rattach_volumes(base_device = nil)
+      dest = base_device
+      if !dest
+        dest = @volumes.map{ |v| v.tags['sdevice']}.min
+      end
+      dest = dest.dup
+
+      @volumes.each do |vol|
+        vol.reload
+        puts "Attaching #{vol.id} to #{dest} on #{vol.tags['dinstance']}"
+        @compute.attach_volume(vol.tags['dinstance'], vol.id, dest)
+        dest.next!
+      end
+    end
+  end
+end
+
+if __FILE__ == $0
+  require 'trollop'
+  require 'ec2_instance_identifier'
+  require 'pp'
+  opts = Trollop::options do
+    opt :hostname, "Hostname tag to use to find the instance", :type => :string, :required => true
+    opt :access_key_id, "Access Key Id for AWS", :type => :string, :required => true
+    opt :secret_access_key, "Secret Access Key for AWS", :type => :string, :required => true
+    opt :date, "Date to restore, use LATEST to take latest data", :type => :string
+    opt :type, "Snapshot type to restore, defaults to snapshot", :type => :string, :default => 'snapshot'
+    opt :target, "Creates volume ready for mounting on instance id. Use special value SELF to restore here", :type => :string
+    opt :first_device, "First device to attach to (default is to use source first device) /dev/sdx", :type => :string
+  end
+    
+  finder = EC2InstanceIdentifier.new(opts[:access_key_id], opts[:secret_access_key])
+  instance_identifier = finder.get_instance(opts[:hostname]).id
+  s = SnapshotRestorer.new(opts[:access_key_id], opts[:secret_access_key])
+
+  # Find this instance snapshots
+  snaps = s.find_snapshots(instance_identifier, opts[:type])
+
+  if ! opts[:date] || !snaps.has_key?(opts[:date])
+    puts "We have found the following snapshot's dates:"
+    snaps.each do |k,v|
+      puts "- #{k} (#{v.length} volume snapshots)"
+    end
+  else
+    puts "Snapshot taken at #{opts[:date]}"
+    snaps[opts[:date]].each do |snapshot|
+      puts "- #{snapshot.id}, #{snapshot.volume_size}GB - #{snapshot.tags['device']}"
+    end
+    if opts[:target]
+      s.snaps = snaps[opts[:date]].map{ |s| s.id }
+      target = opts[:target]
+      target = open("http://169.254.169.254/latest/meta-data/instance-id").read if target == "SELF"
+      puts "Preparing volumes for instance #{target}"
+      s.prepare_volumes(target)
+      # Need to clone, because trollop freeze the variable
+      s.rattach_volumes(opts[:first_device])
+    end
+  end
+
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: mongo-ec2-backup
 version: !ruby/object:Gem::Version
-  version: 0.0.4
+  version: 0.0.5
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-08-06 00:00:00.000000000 Z
+date: 2012-09-20 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: fog
@@ -79,13 +79,14 @@ description:
 email: oct@fotopedia.com
 executables:
 - mongo_lock_and_snapshot
+- ec2_snapshot_restorer
 extensions: []
 extra_rdoc_files: []
 files:
 - lib/ec2-consistent-backup.rb
 - lib/ec2_instance_identifier.rb
-- lib/ec2_snapshot_restorer.rb
 - lib/ec2_volume_snapshoter.rb
+- bin/ec2_snapshot_restorer
 - bin/mongo_lock_and_snapshot
 - Gemfile
 - Gemfile.lock

data/lib/ec2_snapshot_restorer.rb

@@ -1,34 +0,0 @@
-require 'fog'
-require 'open-uri'
-
-
-# 
-class SnapshotRestorer
-  def initialize(aki, sak, snap_ids)
-    @compute = Fog::Compute.new({:provider => 'AWS', :aws_access_key_id => aki, :aws_secret_access_key => sak})
-    @snaps = snap_ids
-    @volumes = []
-  end
-  def restore()
-    @snaps.each do | resource_id |
-      snap = @compute.snapshots.get(resource_id)
-      # Snap have the following tags
-      # application
-      # device
-      # instance_id
-      # date
-      # kind
-
-      t =  snap.tags
-      volume = @compute.volumes.new :snapshot_id => snap.id, :size => snap.volume_size, :availability_zone => 'us-east-1c'
-      @compute.tags.create(:resource_id => volume.id, :key =>"application", :value => NAME_PREFIX)
-      @compute.tags.create(:resource_id => volume.id, :key =>"device", :value => device)
-      @compute.tags.create(:resource_id => volume.id, :key =>"date", :value => ts)
-      @compute.tags.create(:resource_id => volume.id, :key =>"kind", :value => kind)
-      volume.save
-      @volumes << volume
-    end
-  end
-  def connect(instance_id = open("http://169.254.169.254/latest/meta-data/instance-id").read)
-  end
-end