rails-archiver 0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: d8e9e0797cbf13a9816455c8a2386933afb8b413
+  data.tar.gz: 45098734416b669f4011416e0d8d1cec1b859742
+SHA512:
+  metadata.gz: 5067af9aaa1d6a0c6986da09562b95332ea258e7053fc669f6ef5817a3115cc5d43e4202f3de266d31d8bbf842b21e490cd3e23e4f19bd699caa908bb1a056f8
+  data.tar.gz: e93df4a91c299d70d7e708968aa40cb02a364d88149b73cf12c1c0e0e03d8059a3aab63bd51c63c4f616b884f6774697fe13692b483dd44edc1c853068827479

data/README.md

@@ -0,0 +1,58 @@
+# rails-archiver
+**This project is currently in beta.**
+
+This project allows you to archive an entire tree of a model and restore it 
+back. This is useful for cases where there is one central "work item"
+which has many tangential tables associated with it. Rather than having a policy 
+to archive each table after a specific amount of time, or to manually
+find all related items and archive them, this will search through the
+associations of the model, archive them all at once, and allow to restore them.
+
+You can also use this to back up the full model and restore it, e.g. on
+a development machine or another environment. This allows you to easily
+"package up" related data.
+
+The intended usage is to leave the actual work item where it is so that
+you can actually figure out what's available and how to access the associated
+tables - the assumption is that all the associations are what's taking up
+the room in your database. You can inherit from the base `Archiver` class
+if you want to change this behavior.
+
+# Usage
+
+There are two central classes, `Archiver` and `Unarchiver`. Each of them
+take a "Transport" which dictates how to store and retrieve the archive.
+A sample which uses the AWS SDK to save the archive to S3 is provided.
+
+    archiver = RailsArchiver::Archiver.new(my_model, 
+    :transport => :s3, 
+    :delete_records => true)
+    
+    archiver.transport.configure(
+      :bucket_name => 'my_bucket',
+      :base_path => '/path/to/directory')
+    archiver.archive
+    
+    unarchiver = RailsArchiver::Unarchiver.new(my_model)
+    
+## Special attributes
+
+If the model has an attribute called "archived", it will automatically be set
+to true when it's been archived, and false once it's been unarchived. In 
+addition, if using the S3 transport, it will also look for an attribute
+called `archived_s3_key` and set it to the location of the archive.
+
+## Deciding what to archive
+
+By default, the archiver will include all associations which are:
+1) dependent -> `destroy` or `delete_all`
+2) `has_many` or `has_one`
+
+You can change this behavior by subclassing the archiver class and overriding
+the `get_associations` method.
+
+## Compatibility
+
+Currently this project has been tested with Rails 3.0. However, since it uses
+fairly basic Rails model methods, it should be compatible with Rails 4 and 5
+as well.

data/lib/rails-archiver/archiver.rb

@@ -0,0 +1,161 @@
+require 'tmpdir'
+# Takes a database model and:
+# 1) Visits all dependent associations
+# 2) Saves everything in one giant JSON hash
+# 3) Uploads the hash as configured
+# 4) Deletes all current records from the database
+# 5) Marks model as archived
+module RailsArchiver
+  class Archiver
+
+    # Hash which determines equality solely based on the id key.
+    class IDHash < Hash
+      def ==(other)
+        self[:id] == other[:id]
+      end
+    end
+
+    attr_accessor :transport
+
+    # Create a new Archiver with the given model.
+    # @param model [ActiveRecord::Base] the model to archive or unarchive.
+    # @param options [Hash]
+    #   * logger [Logger]
+    #   * transport [Sybmol] :in_memory or :s3 right now
+    #   * delete_records [Boolean] whether or not we should delete existing
+    #     records
+    def initialize(model, options={})
+      @model = model
+      @logger = options.delete(:logger) || ::Logger.new(STDOUT)
+      @hash = {}
+      self.transport = _get_transport(options.delete(:transport) || :in_memory)
+      @options = options
+      # hash of table name -> IDs to delete in that table
+      @ids_to_delete = {}
+    end
+
+    # Archive a model.
+    # @return [Hash] the hash that was archived.
+    def archive
+      @logger.info("Starting archive of #{@model.class.name} #{@model.id}")
+      @hash = {}
+      _visit_association(@model)
+      @logger.info('Completed loading data')
+      @transport.store_archive(@hash)
+      if @model.attribute_names.include?('archived')
+        @model.update_attribute(:archived, true)
+      end
+      @logger.info('Deleting rows')
+      _delete_records if @options[:delete_records]
+      @logger.info('All records deleted')
+      @hash
+    end
+
+    # Returns a single object in the database represented as a hash.
+    # Does not account for any associations, only prints out the columns
+    # associated with the object as they relate to the current schema.
+    # Can be extended but should not be overridden or called explicitly.
+    # @param node [ActiveRecord::Base] an object that inherits from AR::Base
+    # @return [Hash]
+    def visit(node)
+      return {} unless node.class.respond_to?(:column_names)
+       if @options[:delete_records] && node != @model
+        @ids_to_delete[node.class.table_name] ||= Set.new
+        @ids_to_delete[node.class.table_name] << node.id
+      end
+      IDHash[
+        node.class.column_names.select do |cn|
+          next unless node.respond_to?(cn)
+          # Only export columns that we actually have data for
+          !node[cn].nil?
+        end.map do |cn|
+          [cn.to_sym, node[cn]]
+        end
+      ]
+    end
+
+    # Delete rows from a table. Can be used in #delete_records.
+    # @param table [String] the table name.
+    # @param ids [Array<Integer>] the IDs to delete.
+    def delete_from_table(table, ids)
+      return if ids.blank?
+      @logger.info("Deleting #{ids.size} records from #{table}")
+      groups = ids.to_a.in_groups_of(10000)
+      groups.each_with_index do |group, i|
+        sleep(0.5) if i > 0 # throttle so we don't kill the DB
+        delete_query = <<-SQL
+          DELETE FROM `#{table}` WHERE `id` IN (#{group.compact.join(',')})
+        SQL
+        ActiveRecord::Base.connection.delete(delete_query)
+      end
+
+      @logger.info("Finished deleting from #{table}")
+    end
+
+    protected
+
+    # Callback that runs after deletion is finished.
+    def after_delete
+    end
+
+    # Indicate which associations to retrieve from the given model.
+    # @param node [ActiveRecord::Base]
+    def get_associations(node)
+      node.class.reflect_on_all_associations.select do |assoc|
+        [:destroy, :delete_all].include?(assoc.options[:dependent]) &&
+          [:has_many, :has_one].include?(assoc.macro)
+      end
+    end
+
+    private
+
+    # Delete the records corresponding to the model.
+    def _delete_records
+      @ids_to_delete.each do |table, ids|
+        delete_from_table(table, ids)
+      end
+    end
+
+    # @param symbol_or_object [Symbol|RailsArchiver::Transport::Base]
+    # @return [RailsArchiver::Transport::Base]
+    def _get_transport(symbol_or_object)
+      if symbol_or_object.is_a?(Symbol)
+        klass = if symbol_or_object.present?
+                  "RailsArchiver::Transport::#{symbol_or_object.to_s.classify}".constantize
+                  else
+                    Transport::InMemory
+                end
+        klass.new(@model, @logger)
+      else
+        symbol_or_object
+      end
+    end
+
+    # Used to visit an association, and recursively calls down to
+    # all child objects through all other allowed associations.
+    # @param node [ActiveRecord::Base|Array<ActiveRecord::Base>]
+    #    any object(s) that inherits from ActiveRecord::Base
+    def _visit_association(node)
+      return if node.blank?
+      if node.respond_to?(:each) # e.g. a list of nodes from a has_many association
+        node.each { |n| _visit_association(n) }
+      else
+        class_name = node.class.name
+        @hash[class_name] ||= Set.new
+        @hash[class_name] << visit(node)
+        get_associations(node).each do |assoc|
+          @logger.debug("Visiting #{assoc.name}")
+          new_nodes = node.send(assoc.name)
+          next if new_nodes.blank?
+
+          if new_nodes.respond_to?(:each)
+            new_nodes.each { |n| _visit_association(n) }
+          else
+            _visit_association(new_nodes)
+          end
+        end
+
+      end
+    end
+  end
+end

data/lib/rails-archiver/transport/base.rb

@@ -0,0 +1,37 @@
+# Abstract class that represents a way to store and retrieve the generated
+# JSON object.
+module RailsArchiver
+  module Transport
+    class Base
+
+      # @param model [ActiveRecord::Base] the model we will be working with.
+      def initialize(model, logger=nil)
+        @model = model
+        @options = {}
+        @logger = logger || ::Logger.new(STDOUT)
+      end
+
+      # @param options [Hash] A set of options to work with.
+      def configure(options)
+        @options = options
+      end
+
+      # To be implemented by subclasses. Store the archive somewhere to be retrieved
+      # later. You should also be storing the location somewhere such as on the
+      # model. Use @model to reference it.
+      # @param hash [Hash] the hash to store. Generally you'll want to use
+      # .to_json on it.
+      def store_archive(hash)
+        raise NotImplementedError
+      end
+
+      # To be implemented by subclasses. Retrieve the archive that was previously
+      # created.
+      # @return [Hash] the retrieved hash.
+      def retrieve_archive
+        raise NotImplementedError
+      end
+
+    end
+  end
+end

data/lib/rails-archiver/transport/in_memory.rb

@@ -0,0 +1,17 @@
+# Transport that just stores and retrieves the hash in memory.
+module RailsArchiver
+  module Transport
+    class InMemory < Base
+
+      def store_archive(json)
+        @options[:json] = json
+        'some-key-here'
+      end
+
+      def retrieve_archive
+        @options[:json]
+      end
+
+    end
+  end
+end

data/lib/rails-archiver/transport/s3.rb

@@ -0,0 +1,73 @@
+require 'aws-sdk'
+require 'securerandom'
+# Transport that stores to S3. Uses an archived_s3_key attribute.
+module RailsArchiver
+  module Transport
+    class S3 < Base
+
+      def s3_client
+        option_hash = @options[:region] ? {:region => @options[:region]} : {}
+        Aws::S3::Client.new(option_hash)
+      end
+
+      # Gzips the file, returns the gzipped filename
+      def gzip(filename)
+        output = `gzip --force #{filename.shellescape} 2>&1`
+
+        raise output if $?.exitstatus != 0
+
+        "#{filename}.gz"
+      end
+
+      def gunzip(filename)
+        output = `gunzip --force #{filename.shellescape} 2>&1`
+
+        raise output if $?.exitstatus != 0
+      end
+
+      def store_archive(hash)
+        json = hash.to_json
+        file_path = "#{@model.id}_#{SecureRandom.hex(8)}.json"
+        s3_key = "#{@options[:base_path]}/#{file_path}.gz"
+        Dir.mktmpdir do |dir|
+          json_filename = "#{dir}/#{file_path}"
+          @logger.info('Writing hash to JSON')
+          File.write(json_filename, json)
+          @logger.info('Zipping file')
+          filename = gzip(json_filename)
+          @logger.info("Uploading file to #{s3_key}")
+          _save_archive_to_s3(s3_key, filename)
+        end
+        s3_key
+        @model.update_attribute(:archived_s3_key, s3_key)
+      end
+
+      def retrieve_archive
+        Dir.mktmpdir do |dir|
+          filename = "#{dir}/#{@model.id}.json"
+          _get_archive_from_s3(@model.archived_s3_key, "#{filename}.gz")
+          @logger.info('Unzipping file')
+          gunzip("#{filename}.gz")
+          @logger.info('Parsing JSON')
+          JSON.parse(File.read(filename))
+        end
+      end
+
+      private
+
+      def _get_archive_from_s3(s3_key, filename)
+        s3_client.get_object(
+          :response_target => filename,
+          :bucket => @options[:bucket_name],
+          :key => s3_key)
+      end
+
+      def _save_archive_to_s3(s3_key, filename)
+          s3_client.put_object(:bucket => @options[:bucket_name],
+                        :key => s3_key,
+                        :body => File.open(filename))
+
+      end
+    end
+  end
+end

data/lib/rails-archiver/unarchiver.rb

@@ -0,0 +1,125 @@
+# Class that loads a tree hash of objects representing ActiveRecord classes.
+# We will use the models in the codebase to determine how to import them.
+
+require 'activerecord-import'
+
+module RailsArchiver
+  class Unarchiver
+
+    class ImportError < StandardError; end
+
+    attr_accessor :errors, :transport
+
+    # @param model [ActiveRecord::Base]
+    # @param options [Hash]
+    #   * logger [Logger]
+    #   * new_copy [Boolean] if true, create all new objects instead of
+    #                        replacing existing ones.
+    #   * crash_on_errors [Boolean] if true, do not do any imports if any
+    #                               models' validations failed.
+    def initialize(model, options={})
+      @model = model
+      @logger = options.delete(:logger) || Logger.new(STDOUT)
+      @options = options
+      # Transport for downloading
+      self.transport = _get_transport(options.delete(:transport) || :in_memory)
+      self.errors = []
+    end
+
+    # Unarchive a model.
+    def unarchive
+      @errors = []
+      @logger.info('Downloading JSON file')
+      hash = @transport.retrieve_archive
+      @logger.info("Loading #{@model.class.name}")
+      load_classes(hash)
+      @model.reload
+      if @model.attribute_names.include?('archived')
+        @model.update_attribute(:archived, false)
+      end
+      @logger.info("#{@model.class.name} load complete!")
+    end
+
+    # Load a list of general classes that were saved as JSON.
+    # @param hash [Hash]
+    def load_classes(hash)
+      full_hash = hash.with_indifferent_access
+      full_hash.each do |key, vals|
+        save_models(key.constantize, vals)
+      end
+      if @options[:crash_on_errors] && self.errors.any?
+        raise ImportError.new("Errors occurred during load - please see 'errors' method for more details")
+      end
+    end
+
+    # Save all models into memory in the given hash on the given class.
+    # @param klass [Class] the starting class.
+    # @param hashes [Array<Hash<] the object hashes to import.
+    def save_models(klass, hashes)
+      models = hashes.map { |hash| init_model(klass, hash) }
+      import_objects(klass, models)
+    end
+
+    # Import saved objects.
+    # @param klass [Class]
+    # @param models [Array<ActiveRecord::Base>]
+    def import_objects(klass, models)
+      cols_to_update = klass.column_names - [klass.primary_key]
+      # check other unique indexes
+      indexes = ActiveRecord::Base.connection.indexes(klass.table_name).
+        select(&:unique)
+      indexes.each { |index| cols_to_update -= index.columns }
+      options = { :validate => false, :timestamps => false,
+                  :on_duplicate_key_update => cols_to_update }
+
+      @logger.info("Importing #{models.length} for #{klass.name}")
+      models.in_groups_of(1000).each do |group|
+        klass.import(group.compact, options)
+      end
+    rescue => e
+      self.errors << "Error importing class #{klass.name}: #{e.message}"
+    end
+
+    def init_model(klass, hash)
+      attrs = hash.select do |x|
+        klass.column_names.include?(x) && x != klass.primary_key
+      end
+
+      # fix time zone issues
+      klass.columns.each do |col|
+        if col.type == :datetime && attrs[col.name]
+          attrs[col.name] = Time.zone.parse(attrs[col.name])
+        end
+      end
+
+      model = klass.where(klass.primary_key => hash[klass.primary_key]).first
+      if model.nil?
+        model = klass.new
+        model.send(:attributes=, attrs, false)
+        # can't set this in the attribute hash, it'll be overridden. Need
+        # to set it manually.
+        model[klass.primary_key] = hash[klass.primary_key]
+      else
+        model.send(:attributes=, attrs, false)
+      end
+
+      model
+    end
+
+    private
+
+    def _get_transport(symbol_or_object)
+      if symbol_or_object.is_a?(Symbol)
+        klass = if symbol_or_object.present?
+                  "RailsArchiver::Transport::#{symbol_or_object.to_s.classify}".constantize
+                  else
+                    Transport::InMemory
+                end
+        klass.new(@model, @logger)
+      else
+        symbol_or_object
+      end
+    end
+
+  end
+end

data/lib/rails-archiver.rb

@@ -0,0 +1,6 @@
+require 'rails-archiver/transport/base.rb'
+require 'rails-archiver/transport/in_memory.rb'
+require 'rails-archiver/transport/s3.rb'
+
+require 'rails-archiver/archiver.rb'
+require 'rails-archiver/unarchiver.rb'

data/rails-archiver.gemspec

@@ -0,0 +1,18 @@
+Gem::Specification.new do |s|
+  s.name         = 'rails-archiver'
+  s.require_paths = %w(. lib lib/rails-archiver)
+  s.version      = '0.1'
+  s.date         = '2016-03-19'
+  s.summary      = 'Fully archive a Rails model'
+  s.description  = <<-EOF
+EOF
+  s.authors      = ['Daniel Orner']
+  s.email        = 'daniel.orner@wishabi.com'
+  s.files        = `git ls-files`.split($/)
+  s.homepage     = 'https://github.com/dmorner/rails-archiver'
+  s.license       = 'MIT'
+
+  s.add_dependency 'rails', '>= 3.0'
+  s.add_dependency 'aws-sdk', '~> 2.6'
+
+end

metadata

@@ -0,0 +1,81 @@
+--- !ruby/object:Gem::Specification
+name: rails-archiver
+version: !ruby/object:Gem::Version
+  version: '0.1'
+platform: ruby
+authors:
+- Daniel Orner
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2016-03-19 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: rails
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '3.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '3.0'
+- !ruby/object:Gem::Dependency
+  name: aws-sdk
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.6'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.6'
+description: ''
+email: daniel.orner@wishabi.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- README.md
+- lib/rails-archiver.rb
+- lib/rails-archiver/archiver.rb
+- lib/rails-archiver/transport/base.rb
+- lib/rails-archiver/transport/in_memory.rb
+- lib/rails-archiver/transport/s3.rb
+- lib/rails-archiver/unarchiver.rb
+- rails-archiver.gemspec
+homepage: https://github.com/dmorner/rails-archiver
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- "."
+- lib
+- lib/rails-archiver
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.5.2
+signing_key: 
+specification_version: 4
+summary: Fully archive a Rails model
+test_files: []