aws_xregion_sync 0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 62a8ca4d29553c5aa20118465c339057c0f9545a
+  data.tar.gz: 5caabbc8f7f3b39c78f98e96bf8e9af18462e1c9
+SHA512:
+  metadata.gz: 6bae78d1fdc1ddb8be11e51f42fe3830d0403a21e313b9db797780794cee96933a9eed395dc95de8b1010f314138cf80d8a8f1249b67ae8a44ca4420a7da3c6c
+  data.tar.gz: 7a640afc58a6b5c43669979258380cf7c89271bc846f53289df6168d740d79cdc17b5e3170bc77b77f2464f017d2a632b6fbacee90bc97a7cb624eeffa845060

data/LICENSE.txt

@@ -0,0 +1,20 @@
+The MIT License (MIT)
+
+Copyright (c) 2013 Vandegrift Forwarding Company Inc. 
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
+the Software, and to permit persons to whom the Software is furnished to do so,
+subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
+FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,138 @@
+aws_xregion_sync
+================
+
+AWS Cross-Region Sync is a simple tool to help provide for easy disaster recovery by directly syncing AWS resources across AWS regions.
+
+At this time only EC2 AMI and RDS Automated Snapshot syncing is supported.
+
+## How To
+
+Configuring and running AWS X Region Sync is done via a simple YAML configuration file.  The easiest way to show how to use the system is provide an example
+config file and then walk through the options.
+
+Assume the following config data is found in the file '/my/config.yaml':
+
+```
+{
+  
+  sync_my_web_app: {
+    sync_type: "ec2_ami",
+    source_region: "us-east-1",
+    destination_region: "us-west-1",
+    ami_owner: "12345678910123",
+    sync_identifier: "Web Application",
+    # http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-DescribeImages.html
+    filters: ["tag:Environment=Production", "tag-value=Sync"]
+  },
+  sync_my_database: {
+    sync_type: "rds_automated_snapshot",
+    source_region: "us-east-1",
+    destination_region: "us-west-1",
+    db_instance: "mydatabase",
+    max_snapshots_to_retain: 5,
+    aws_client_config: {
+      aws_access_key: "my_other_access_key",
+      aws_secret_key: "my_other_secret_key",
+      aws_account_id: "4321-4321-4321"
+    }
+  },
+  aws_client_config: {
+    aws_access_key: "my_access_key",
+    aws_secret_key: "my_secret_key",
+  }
+}
+```
+
+Each YAML key that starts with 'sync_' defines a distinct job to sync 1 particular resource.  Lets examine each of these sync types now.
+
+### ec2_ami
+The 'sync_my_web_app' job uses the 'ec2_ami' type which locates a single AMI instance associated with the given account credentials and will copy the image
+and all resource tags associated with the image to the defined 'destination_region'.  If the source image has already been copied to the destination region 
+this job will be a no-op.  The process utilizes a single resource tag as means of tracking if the image has already been synced and will NOT repeatedly sync 
+the same image to the same region if it can determine the destination region already contains a copy of the source image.
+
+Copying the image may take quite some time, depending on the size of the AMI.  The sync job does not block while the image is being copied and will not report
+the final status of the AWS copy task.  It is assumed that the AMI copy will eventually complete.
+
+#### ec2_ami Configuration
+
+The following configuration options are available for 'ec2_ami' sync jobs (star'ed options are required):
+
+- source_region * - The name of the AWS region the source EC2 AMI will be found in.
+- destination_region * - The name of the AWS region the source EC2 AMI should be copied to.
+- ami_owner - The AWS owner id of the AMI.  If left blank, the account associated with the AWS client credentials is used.
+- sync_identifier - If given, a resource tag named 'Sync-Identifier' with the provided value will expected to be associated with the source AMI.  This is probably the simplest means of identifying which AMI to sync.
+- filters - An array of String filter values that can be used to further filter AMI options down to a single source AMI.  Multiple filters are ANDed togther.  See the 'filter' parameter here for further explanation of available filter values: http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-DescribeImages.html
+
+In general, the combination of ami_owner, sync_identifier, and filter options MUST narrow down the list of AMI images to a SINGLE AMI.  If they do not, the sync job will be aborted.
+
+NOTE: Under the covers, the sync task is performed by the AWS <a href="http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-CopyImage.html">ec2-copy-image</a> API method.
+
+### rds_automated_snapshot
+The 'sync_my_database' job uses the 'rds_automated_snapshot' type which locates the newest automated rds snapshot associated with the given 'db_instance' and utilizes
+the AWS copy_snapshot functionality to copy the snapshot to the destination region.  If the source snapshot is determined to already have been synced to the destination
+snapshot the job will be a no-op.  The process utilizes the snapshot's created at attribute combined with a resource tag on destination snapshots to determine if
+the snapshot has already been synced.
+
+Copying the snapshot may take quite some time, depending on the size of the snapshot and amount of time since the last snapshot has been synced.  
+The sync job does not block while the snapshot is being copied and will not report the final status of the AWS copy task. 
+It is assumed that the snapshot copy will eventually complete.
+
+NOTE: Under the covers, the sync task is performed by the AWS <a href="http://docs.aws.amazon.com/AmazonRDS/latest/CommandLineReference/CLIReference-cmd-CopyDBSnapshot.html">rds-copy-db-snapshot</a> API method.  Please see it for any further explanation of how the sync is performed.  In particular note that once an initial snapshot
+has been copied only incremental changes are copied between regions, saving both time and bandwidth costs.  Each source automated snapshot that is
+copied will result in a new destination snapshot.  The sync job will retain however many snapshots in the destination region you would like to keep.
+
+#### rds_automated_snapshot Configuration
+
+The following configuration options are available for 'ec2_ami' sync jobs (star'ed options are required):
+
+- source_region * - The name of the AWS region the source RDS Snapshot will be found in.
+- destination_region * - The name of the AWS region the source RDS Snapshot should be copied to.
+- db_instance * - The RDS DB Instance Identifier value for the source database in the source region.
+- max_snapshots_to_retain - The number of snapshots to retain in the destination region.  Defaults to retaining 2 existing ones - which ends up being 3 total including the one in the process of copying.  You'll probably wan't to keep this to at least 1 so that you'll always have at least a single previous snapshot available while the current one copies over to the dest. region.
+- aws_client_config/aws_account_id- The AWS API for RDS snapshot tags requires providing an account number, therefore, the effective aws_client_credentials utilized for an rds snapshot sync can contain an account number.  This value can be found automatically in a rather hacky fashion by examining IAM user ARNs associated with the AWS keys, and this method will be utilized if no aws_account_id is provided.  However, be prepared for this method to fail and to have to provide the account id directly.
+
+### aws_client_config
+
+The 'aws_client_config' values can be defined both at a global level and/or inside each individual sync job.  The values defined at the global level are merged
+together with any values defined at the job level (pretty much exactly as global_config.merge(job_config)).  The resulting config hash comprised of one or both values
+are then passed directly to the AWS SDK.  Because of this, you can provide any acceptable configuration values to the AWS ruby SDK client.
+
+Addtional aws client config properties are:
+
+- aws_account_id - Some sync tasks require the use of a account id to construct ARN's.  In some cases, the account id can be deduced from the access and secret key via an IAM user lookup.  When this is not possible, an error will be raised associated with the sync job and you will need to provide the aws_account_id manually.
+
+## Sample Code
+
+In general, the only class/method your code needs to call is the AwsXRegionSync.run method.  It will return you a collection of AwsXRegionSync::SyncResult objects, 
+one for each sync job contained in your config file.
+
+Here's a really simple, direct example of running the config file above (apologies for verbosity of if statements - easiest most direct way of showing potential return values from the run method):
+```ruby
+require 'aws_xregion_sync'
+results = AwsXRegionSync.run '/my/config.yaml'
+results.each do |result|
+  if result.failed?
+    puts "#{result.name} encountered the following errors:\n#{result.errors.map(&:message).join('\n')}"
+  else
+    # If an image/snapshot was created by the job, created_resource will be populated. 
+    # Resources may not be created in such cases where an EC2 image may already be in sync.
+    if result.created_resource
+      puts "#{result.name} successfully completed and created the AWS resource #{result.created_resource}."
+    else
+      puts "#{result.name} successfully completed without creating any new AWS resources."
+    end
+  end
+end
+
+```
+
+## Contributing
+
+See something you want added or have a bug that needs sqashing?  
+Feel free to open an issue and we'll get back to you OR better yet, fork the project, create a topic branch, code up your awesome change, push to your branch and open a pull request with us.  
+
+License
+-------
+
+aws_xregion_sync is available under the MIT License.  See LICENSE.txt for more information 

data/lib/aws_xregion_sync.rb

@@ -0,0 +1,62 @@
+require 'aws-sdk'
+require 'require_all'
+require_rel 'aws_xregion_sync'
+
+class AwsXRegionSync
+
+  def self.run config_file_path_or_hash
+    do_syncs config_file_path_or_hash, :sync
+  end
+
+  def self.sync_required? config_file_path_or_hash
+    do_syncs config_file_path_or_hash, :sync_required?
+  end
+
+  def self.do_syncs config_file_path_or_hash, job_method
+    job_configs = configure config_file_path_or_hash
+    error_results = create_results_from_config_errors job_configs[:errors]
+    job_syncs = sync job_configs[:jobs], job_method
+
+    job_syncs + error_results
+  end
+  private_class_method :do_syncs
+
+  def self.configure config
+    # Use to_hash since this will allow a broader range of config options to anyone
+    # passing in a config object rather than a filename
+    if config.respond_to? :to_hash
+      Configure.generate_sync_jobs config.to_hash
+    else
+      Configure.configure_from_file config 
+    end
+  end
+  private_class_method :configure
+
+  def self.sync jobs, job_method
+    results = []
+    jobs.each do |job|
+      completed = false
+      errors = nil
+      synced_object_id = nil
+      begin
+        synced_object_id = job.send job_method
+        completed = true
+      rescue Exception => e
+        # Yes, we're trapping everything here.  I'd like all the sync jobs to at least attempt
+        # to run each time sync is called.  If they all bail with something like a memory error
+        # or something else along those lines, then so be it, the error will get raised eventually
+        errors = [e]
+      end
+
+      results << SyncResult.new(job.sync_name, completed, synced_object_id, errors)
+    end
+
+    results
+  end
+  private_class_method :sync
+
+  def self.create_results_from_config_errors errors
+    errors ? errors.map {|job_key, config_errors| SyncResult.new(job_key, false, nil, config_errors)} : []
+  end
+  private_class_method :create_results_from_config_errors
+end

data/lib/aws_xregion_sync/aws_sync.rb

@@ -0,0 +1,119 @@
+class AwsXRegionSync
+  class AwsSync
+
+    attr_reader :config, :sync_name
+
+    def initialize sync_name, config
+      @sync_name = sync_name
+      @config = config
+    end
+
+    def validate_config
+      # We could validate if these are actual valid region names via the AWS client, but for now we'll assume we're actually getting something valid
+      # as long as the option is there - the sync will blow up and give the reason anyway.
+      raise AwsXRegionSyncConfigError, "The #{sync_name} configuration must have a valid 'source_region' value." unless config['source_region'] && config['source_region'].length > 0
+      raise AwsXRegionSyncConfigError, "The #{sync_name} configuration must have a valid 'destination_region' value." unless config['destination_region'] && config['destination_region'].length > 0
+
+      if config['filters']
+        raise AwsXRegionSyncConfigError, "The #{sync_name} configuration 'filters' value must be an Enumerable object." unless config['filters'] && config['filters'].is_a?(Enumerable)
+        parse_config_filters config['filters']
+      end
+      self
+    end
+
+    def parse_config_filters config_filters
+      # Splits all the given filters on '=' so they can be utilized in a standard describe filter clause
+      filters = []
+      config_filters.each do |cf|
+        split = cf.to_s.split("=")
+        raise AwsXRegionSyncConfigError, "The #{sync_name} configuration 'filters' value '#{cf}' must be of the form filter-field=filter-value." unless split.size == 2
+
+        filters << split
+      end
+      filters
+    end
+
+    def aws_config
+      global = config['aws_client_config']
+      # We want to make sure if we create a new config hash that we're
+      # storing it so any changes made by a caller to the returned hash
+      # are stored off
+      unless global 
+        global = {}
+        config['aws_client_config'] = global
+      end
+
+      global
+    end
+
+    def create_sync_tag region, source_identifier, options = {}
+      options = {timestamp: Time.now}.merge options
+
+      timestamp = options[:timestamp].utc
+      key = "#{sync_tag_indicator}"
+      key += "-#{options[:sync_subtype]}" if options[:sync_subtype]
+      key += "-#{region}"
+      {key: key, value: build_sync_tag_value(source_identifier, timestamp)}
+    end
+
+    def parse_sync_tag_value value
+      # Sync Timestamp (YYYYMMDDHHmm) to second / resource identifier
+      # AWS only allows 10 tags per resource so we're trying to limit the # of tags we consume by combining the timestamp and resource identifier into a single tag
+      values = {}
+      if value && value.length > 0
+        split = value.split(" / ")
+        values[:timestamp] = Time.strptime(split[0], timestamp_format).utc
+        values[:resource_identifier] = split[1]
+      end
+      
+      values
+    end
+
+    def sync_tag_indicator
+      "Sync"
+    end
+
+    def discover_aws_account_id raise_error_if_not_found = true
+      return @aws_account_id if defined?(@aws_account_id)
+
+      aws_account_id = aws_config['aws_account_id']
+      unless aws_account_id
+        aws_account_id = retrieve_aws_account_id
+      end
+      raise AwsXRegionSyncConfigError, "The #{self.sync_name} configuration must provide an 'aws_account_id' option to use for manipulating snapshot tags, unable to retrieve id automatically." if (aws_account_id.nil? || aws_account_id.length == 0) && raise_error_if_not_found
+
+      @aws_account_id = aws_account_id
+    end
+
+    private
+
+      def timestamp_format
+        "%Y%m%d%H%M%S%z"
+      end
+
+      def build_sync_tag_value source_identifier, sync_timestamp
+        "#{sync_timestamp.strftime(timestamp_format)} / #{source_identifier}"
+      end
+
+      def create_iam config
+        #split out for mocking purposes
+        AWS::IAM.new config
+      end
+
+      def retrieve_aws_account_id
+        # We can use the ARN associated with any user to find the account id associated with the access/secret key from the config
+        # This feels like a massive hack, but it's the only automated way I've found to retrieve this information that's needed to construct
+        # resource ARN's for other direct API calls.
+        iam = create_iam aws_config
+        arn = nil
+        iam.users.each(limit: 1) {|u| arn = u.arn}
+
+        account_id = nil
+        if arn && arn =~ /^arn:aws:iam::(\d+):/
+          account_id = $1
+        end
+
+        account_id
+      end
+  end
+end

data/lib/aws_xregion_sync/configure.rb

@@ -0,0 +1,69 @@
+require 'yaml'
+
+class AwsXRegionSync
+  class Configure
+
+    def self.configure_from_file file_path
+      # For now, just assume a yaml config file..we can certainly support json/xml or something more complicated here internally later.
+      generate_sync_jobs load_yaml_config_file file_path
+    end
+
+    def self.generate_sync_jobs config
+      # We need at least one sync key, each sync must have a type indicator, each sync must have a source and destination region
+      sync_keys = config.keys
+      
+      global_aws_config = config['aws_client_config']
+      sync_jobs = []
+      configuration_errors = {}
+      sync_keys.each do |job_key|
+        if job_key.upcase.to_s.start_with? "SYNC_"
+          sync_config = config[job_key]
+          begin
+            sync_jobs << create_sync_job(global_aws_config, job_key, sync_config)
+          rescue => e
+            configuration_errors[job_key] = [e]
+          end
+        end
+      end
+
+      {jobs: sync_jobs, errors: configuration_errors}
+    end
+
+    def self.create_sync_job global_aws_config, job_key, job_config
+      # merge the global and local aws client config settings together, allowing for easy global and per sync config settings 
+      if global_aws_config
+        if job_config['aws_client_config']
+          job_config['aws_client_config'] = global_aws_config.merge job_config['aws_client_config']
+        else
+          # Just want to make a new hash object here
+          job_config['aws_client_config'] = global_aws_config.merge({})
+        end
+      end
+
+      sync_job = nil
+      case job_config['sync_type']
+      when 'ec2_ami'
+        sync_job = Ec2AmiSync
+      when 'rds_automated_snapshot'
+        sync_job = RdsAutomatedSnapshotSync
+      else
+        raise AwsXRegionSyncConfigError, "The #{job_key} configuration 'sync_type' value '#{job_config['sync_type']}' is not a supported AWS Sync type."
+      end
+
+      job = sync_job.new job_key, job_config
+      job.validate_config
+      job
+    end
+    private_class_method :create_sync_job
+
+    def self.load_yaml_config_file config
+      if config.is_a? String
+        YAML.load_file config
+      else
+        YAML.load config
+       end
+    end
+    private_class_method :load_yaml_config_file
+
+  end
+end

data/lib/aws_xregion_sync/ec2_ami_sync.rb

@@ -0,0 +1,115 @@
+class AwsXRegionSync
+  class Ec2AmiSync < AwsSync
+
+    def sync
+      setup = pre_sync_setup
+
+      sync_ec2_image_to_region setup[:destination_region], setup[:source_region], setup[:image]
+    end
+
+    def sync_required?
+      setup = pre_sync_setup
+      needs_sync? setup[:destination_region], setup[:image]
+    end
+
+    def sync_ec2_image_to_region destination_region, source_region, image
+      destination_ami_id = nil
+
+      if needs_sync?(destination_region, image)
+        # At this point, we know the sync has not happened (or the destination image has been removed)
+        # Initiate the image copy command (we're not using client_token to ensure indempotency since we're already just using the Sync- identifier tag to ensure we're not 
+        # copying the image multiple times to the destination region)
+        results = destination_region.client.copy_image source_region: source_region.name, source_image_id: image.id, name: image.name
+
+        destination_ami_id = results[:image_id]
+
+        tag = create_sync_tag destination_region.name, destination_ami_id
+        # We want to log the destination AMI-ID and the time we started the sync back to the source image
+        image.tags[tag[:key]] = tag[:value]
+      end
+      
+      destination_ami_id
+    end
+
+    private 
+
+      def pre_sync_setup 
+        config = {'owner_id'=>'self'}.merge self.config
+
+        filters = parse_filters config['sync_identifier'], config['filters']
+
+        ec2 = ec2_client aws_config
+
+        source_region = validate_region ec2, config['source_region']
+        destination_region = validate_region ec2, config['destination_region']
+
+        # Assemble the filter criteria that should help us to find the specific image we're after
+        ami_query = source_region.images
+        ami_query = ami_query.with_owner(config['owner_id'])
+        
+        filters.each {|f| ami_query = ami_query.filter(f[0], f[1])}
+
+        images = ami_query.to_a
+        if images.size > 1
+          raise AwsXRegionSyncConfigError, "More than one EC2 Image was found with the filter settings for identifier '#{self.sync_name}': #{images.collect{|i| i.id}.join(", ")}."
+        elsif images.size == 0
+          raise AwsXRegionSyncConfigError, "No EC2 Images were found using the filter settings for identifier '#{self.sync_name}'."
+        end
+
+        {destination_region: destination_region, source_region: source_region, image: images[0]}
+      end
+
+      def needs_sync? destination_region, image
+        source_tags = image.tags.to_a
+
+        # Look for a tag indicating the image has been synced to the destination region
+        dest_ami_id = find_destination_ami source_tags, destination_region.name
+        !image_exists(destination_region, dest_ami_id)
+      end 
+
+      def ec2_client aws_client_config
+        AWS::EC2.new(aws_client_config)
+      end
+
+      def image_exists destination_region, ami_id
+        image = ami_id ? destination_region.images[ami_id] : nil
+        !image.nil? && image.exists?
+      rescue
+        false
+      end
+
+      def parse_filters sync_identifier, config_filters
+        filters = []
+        filters << ["tag:#{sync_tag_indicator}-Identifier", sync_identifier] if sync_identifier && sync_identifier.length > 0
+        if config_filters && config_filters.length > 0
+          filters = filters + parse_config_filters(config_filters)
+        end
+        filters
+      end
+
+      def validate_region ec2, region_id
+        region = ec2.regions[region_id]
+
+        # AWS does an actual HTTP query to establish if the region exists, and then doesn't catch any socket errors
+        # if it fails. Since the aws gem directly uses the value you pass for the region as part of the http host it
+        # calls the gem ends up raising a naming exception when you used a bad region name instead of just returning 
+        # false that the region doesn't exist
+        exists = false
+        begin
+          exists = region.exists?  
+        rescue
+        end
+
+        raise AwsXRegionSyncConfigError, "Invalid region code of '#{region_id}' found for identifier '#{self.sync_name}'.  It either does not exist or the given credentials cannot access it." unless exists
+        region
+      end
+
+      def find_destination_ami tags, region
+        # We're expecting the tags to be like [["Key1", "Value1"], ["Key2", "Value2"]]
+        # which is how to_a on a TagCollection object retutns the values
+        key = create_sync_tag(region, nil)[:key]
+        tag_value = parse_sync_tag_value(tags.detect(->(){[]}) {|tag| tag[0] == key}[1])
+        tag_value[:resource_identifier]
+      end
+  end
+end

data/lib/aws_xregion_sync/errors.rb

@@ -0,0 +1,7 @@
+class AwsXRegionSync
+  class AwsXRegionSyncError < StandardError
+  end
+
+  class AwsXRegionSyncConfigError < AwsXRegionSyncError
+  end
+end

data/lib/aws_xregion_sync/rds_automated_snapshot_sync.rb

@@ -0,0 +1,202 @@
+class AwsXRegionSync
+  class RdsAutomatedSnapshotSync < AwsSync
+
+    def validate_config
+      raise AwsXRegionSyncConfigError, "The #{self.sync_name} configuration must provide a 'db_instance' option to use to locate automated snapshots." unless config['db_instance'] && config['db_instance'].length > 0
+      # This call will raise if there's an invalid value..so just call it right now
+      max_snapshots
+      super
+    end
+
+    def sync
+      setup = pre_sync_setup
+      sync_snapshot_to_region setup[:source_region], setup[:destination_region], setup[:snapshot]
+    end
+
+    def sync_required?
+      setup = pre_sync_setup
+      sync = needs_sync? setup[:destination_region], setup[:source_region], setup[:snapshot], discover_aws_account_id
+      sync[:sync_required]
+    end
+
+    def sync_snapshot_to_region source_region, destination_region, source_snapshot
+      # The way automated snapshot copying seems to work is as long as you're copying snapshot from the same db instance between regions
+      # every snapshot you copy from the source region after the first one is simply just an incremental copy.  Because of this we
+      # should copy the latest snapshot
+
+      # This first thing we should do is look for a sync tag in the destination snapshot (if present) and see we've already synced this
+      # snapshot-id (or if the id is outdated)
+      aws_account_id = discover_aws_account_id
+      
+      sync = needs_sync?(destination_region, source_region, source_snapshot, aws_account_id)
+      if sync[:sync_required]
+        sr = region(source_region)
+        result = destination_region.client.copy_db_snapshot source_db_snapshot_identifier: arn(sr, aws_account_id, source_snapshot.id), target_db_snapshot_identifier: sanitize_snapshot_id(source_snapshot.id)
+
+        if result[:db_snapshot_identifier]
+          destination_snapshot_id = result[:db_snapshot_identifier]
+
+          # Move the source snapshot's created timestamp and id to the destination snapshot's sync tag so that we know the last time
+          # the snapshot was synced and what region it was synced from.  Also, log the sync against the source too so we have tangible evidence
+          # that it was synced just by looking at it as well.
+          dr = region(destination_region)
+          
+          dest_sync_tag = create_sync_tag sr, source_snapshot.id, timestamp: source_snapshot.created_at, sync_subtype: "From"
+          source_sync_tag = create_sync_tag dr, destination_snapshot_id, timestamp: source_snapshot.created_at
+
+          destination_region.client.add_tags_to_resource resource_name: arn(dr, aws_account_id, result[:db_snapshot_identifier]), tags: [dest_sync_tag]
+          source_region.client.add_tags_to_resource resource_name: arn(sr, aws_account_id, source_snapshot.id), tags: [source_sync_tag]
+
+          # We'll now clean up older snapshots if necessary
+          cleanup_old_snapshots sync[:snapshots], max_snapshots, destination_region
+        end
+      end
+
+      destination_snapshot_id
+    end
+
+    private
+
+      def pre_sync_setup
+        # There doesn't appear to be a direct way of actually obtaining
+        # a region reference after initializing the rds client - like there is for the s3 client
+        # so we'll just create multiple clients.
+        source_rds = rds_client config['source_region'], config['db_instance']
+        destination_rds = rds_client config['destination_region'], config['db_instance']
+
+        instance = source_rds.db_instances[config['db_instance']]
+        raise AwsXRegionSyncConfigError, "No DB Instance with identifier '#{config['db_instance']}' is available for these credentials in region #{config['db_instance']}." unless instance
+
+        # Find all the snapshots
+        snapshots = instance.snapshots.with_type('automated').to_a
+        raise AwsXRegionSyncConfigError, "No automated snapshots for db '#{config['db_instance']}' are available for these credentials in region #{config['source_region']}." unless snapshots.size > 0
+
+        # Use the created_at attribute of the snapshot to find the newest one
+        newest_snapshot = extract_newest_snapshot snapshots
+
+        {source_region: source_rds, destination_region: destination_rds, snapshot: newest_snapshot}
+      end
+
+      def needs_sync? destination_region, source_region, source_snapshot, aws_account_id
+        destination_snapshots = retrieve_destination_snapshots_for_instance destination_region, region(source_region), source_snapshot.db_instance.id, aws_account_id
+        sync_needed = destination_snapshots.size == 0 || snapshot_needs_copying(destination_snapshots, source_snapshot)
+
+        {sync_required: sync_needed, snapshots: destination_snapshots}
+      end
+
+      def rds_client region, db_instance
+        # There doesn't appear to be a direct way of actually obtaining
+        # a region reference after initializing an rds client object (unlike w/ the EC2 API)
+        rds = make_rds aws_config.merge({region: region})
+        # Just force an API call to validate the region setting.  Use the db instance call which we'll use later and, I believe, is cached
+        # so there should be no penalty for calling it here (since we use it pretty much directly after this call anyway)
+        rds.db_instances[db_instance]
+        rds
+      rescue
+        raise AwsXRegionSyncConfigError, "Region '#{region}' is invalid.  It either does not exist or the given credentials cannot access it."
+      end
+
+      def make_rds config
+        # purely for mocking
+        AWS::RDS.new config
+      end
+
+      def extract_newest_snapshot snapshots
+        # Grab the snapshot created at time for snapshot as the hash key, then we can sort on the keys and extract the last one from that sorted array
+        snapshot_values = {}
+        snapshots.each {|s| snapshot_values[s.created_at] = s}
+        snapshot_values[snapshot_values.keys.sort[-1]]
+      end
+
+      def arn region, account_number, snapshot_id
+        # Strip all non-decimal characters from account numbers
+        account_number = account_number.gsub(/\D/, "")
+        "arn:aws:rds:#{region}:#{account_number}:snapshot:#{snapshot_id}"
+      end
+
+      def region client
+        client.config.region
+      end
+
+      def snapshot_needs_copying destination_snapshots, source_snapshot
+        # The snapshot array is already sorted, so the last value from the array is the newest one
+        newest_snapshot_hash = destination_snapshots.last
+        source_snapshot.created_at.to_i > newest_snapshot_hash[:sync_timestamp].to_i
+      end
+
+      def tags_for_snapshot rds, account_number, snapshot_id
+        region_client = rds.client
+        region_name = region(region_client)
+
+        aws_resource = arn(region_name, account_number, snapshot_id)
+        result = region_client.list_tags_for_resource resource_name: arn(region_name, account_number, snapshot_id)
+        result[:tag_list]
+      end
+
+      def retrieve_destination_snapshots_for_instance destination_region, source_region_name, db_instance, account_number
+        snapshots = destination_region.db_instances[db_instance].snapshots.with_type('manual').to_a
+
+        # order them newest to oldest based on their sync tags
+        snapshot_list = []
+        snapshots.each do |s|
+          tags = tags_for_snapshot(destination_region, account_number, s.id)
+
+          sync_tag = find_sync_tag tags, source_region_name, db_instance
+          # Ignore any snapshot that doesn't have a sync tag, these are likely actual user initiated manual copies and we don't want to mess with them
+          next unless sync_tag
+
+          split_sync_value = parse_sync_tag_value(sync_tag[:value]) if sync_tag
+          snapshot_list << {snapshot: s, tags: tags, :sync_timestamp => split_sync_value[:timestamp], :sync_tag=>sync_tag}
+        end
+
+        snapshot_list.sort_by {|snapshot| snapshot[:sync_timestamp]}
+      end
+
+      def make_to_snapshot_sync_tag_key source_region
+        create_sync_tag(source_region, nil, sync_subtype: "From")[:key]
+      end
+
+      def find_sync_tag tags, source_region_name, db_instance
+        sync_key = make_to_snapshot_sync_tag_key source_region_name
+
+        tags.each do |t|
+          return t if t[:key] == sync_key
+        end
+
+        nil
+      end
+
+      def sanitize_snapshot_id id
+        # The official "rules" for identifiers are: 
+        # Identifiers must begin with a letter; must contain only ASCII letters, digits, and hyphens;"
+        # and must not end with a hyphen or contain two consecutive hyphens
+        # Ids for automated snapshots appear to look like "rds:db_instance-YYYY-mm-dd-HH-MM"..that hyphen is invalid
+        # For now, just translate any : to -
+        id.gsub(":", "-")
+      end
+
+      def cleanup_old_snapshots destination_snapshots, max_snapshots_to_retain, destination_region
+        if destination_snapshots.size > max_snapshots_to_retain
+          # Grab the first max_snapshots # of destination_snapshots (since they're sorted already in ascending order) and just delete them
+          snapshots_to_delete = destination_snapshots.take(destination_snapshots.size - max_snapshots_to_retain)
+
+          snapshots_to_delete.each do |snapshot|
+            destination_region.client.delete_db_snapshot db_snapshot_identifier: snapshot[:snapshot].id
+          end
+        end
+      end
+
+      def max_snapshots
+        retain = nil
+        if config['max_snapshots_to_retain']
+          retain = Integer(config['max_snapshots_to_retain']) 
+        else
+          retain = 2
+        end
+
+        retain
+      rescue 
+        raise AwsXRegionSyncConfigError, "The #{self.sync_name} configuration must provide a valid 'max_snapshots_to_retain' option. '#{config['max_snapshots_to_retain']}' is not valid."
+      end
+  end
+end

data/lib/aws_xregion_sync/sync_result.rb

@@ -0,0 +1,22 @@
+class AwsXRegionSync
+  class SyncResult
+
+    attr_reader :name, :completed, :created_resource, :errors
+
+    def initialize name, completed, created_resource, errors = []
+      @name = name
+      @completed = completed
+      @created_resource = created_resource
+      @errors = errors ? errors : []
+    end
+
+
+    def failed?
+      !completed
+    end
+
+    def sync_required?
+      completed && created_resource == true
+    end
+  end
+end

data/lib/aws_xregion_sync/version.rb

@@ -0,0 +1,3 @@
+class AwsXRegionSync
+  VERSION ||= "0.1"
+end

metadata

@@ -0,0 +1,97 @@
+--- !ruby/object:Gem::Specification
+name: aws_xregion_sync
+version: !ruby/object:Gem::Version
+  version: '0.1'
+platform: ruby
+authors:
+- Vandegrift Forwarding Company
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2013-12-12 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: aws-sdk
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '1.17'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '1.17'
+- !ruby/object:Gem::Dependency
+  name: require_all
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+description: Sync EC2 AMIs and RDS Snapshots across AWS regions as part of your Disaster
+  Recovery planning.
+email:
+- it-admin@vandegriftinc.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- lib/aws_xregion_sync.rb
+- lib/aws_xregion_sync/aws_sync.rb
+- lib/aws_xregion_sync/ec2_ami_sync.rb
+- lib/aws_xregion_sync/rds_automated_snapshot_sync.rb
+- lib/aws_xregion_sync/configure.rb
+- lib/aws_xregion_sync/errors.rb
+- lib/aws_xregion_sync/sync_result.rb
+- lib/aws_xregion_sync/version.rb
+- LICENSE.txt
+- README.md
+homepage: http://github.com/Vandegrift/aws_xregion_sync
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '2'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '1.5'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.0.7
+signing_key: 
+specification_version: 4
+summary: Simple tool to help sync Amazon resources across regions.
+test_files: []