importer 0.3.2 → 0.4.0

data/README.rdoc

@@ -12,7 +12,6 @@ Docs: http://rdoc.info/projects/szajbus/importer
 
 * can import from XML and CSV formats, but it's possible to add custom parsers
 * reports how many new objects got imported, how many objects was modified and how many objects were invalid
-* can save reports to the database, along with errors for later inspection
 
 == Installation
 
@@ -24,11 +23,6 @@ Or the plugin
 
   script/plugin install git://github.com/szajbus/importer.git
 
-Generate the migrations and run them (this step may not be necessary, read below)
-
-  script/generate importer import
-  rake db:migrate
-
 Add to your model
 
   class Product < ActiveRecord::Base
@@ -39,21 +33,17 @@ And start importing
 
   Product.import(path_to_xml_or_csv_file)
 
-This will parse the file and import all products there are defined in it. Import summary will be saved in imports table, it will tell you how many products were created, modified or invalid. Exact information about each product (detected attributes and errors) will be saved in imported_objects table.
-
-If you don't want to save summaries to database you can force the importer to do a simple import with:
-
-  Product.import(path_to_xml_or_csv_file, :import => Importer::Import::Simple.create)
+This will parse the file and import all products there are defined in it. This will return import summary which will tell you how many products were created, modified or invalid. Exact information about each product (detected attributes and errors) will be available in summary too.
 
 == Customization
 
 You can create your own parser to import from sources other than XML or CSV files. Check the implementation of one of existing parsers to find out how to write your own. Then just pass parser class to import method:
 
-  Product.import(path_to_xml_or_csv_file, :parser => YourCustomParser)
+  Product.import(path_to_file, :parser => CustomParserClass)
 
-You can also create your custom Import and ImportedObject classes if you need the import summaries in other way than those currently offered (ActiveRecord which stores summaries in database and Simple which does not store them at all). Again check the current implementations to get the idea. To force the importer to use it:
+You can also create your custom versions of Import and ImportedObject classes. A possible alternative version could be ActiveRecord Import and ImportedObject models that would save import summary to database for later inspection. Check the rdocs for these classes for more information. You force the importer to use a custom Import class with:
 
-  Product.import(path_to_xml_or_csv_file, :import => YourCustomImport.new)
+  Product.import(path_to_file, :import => CustomImportClass)
 
 == Note on Patches/Pull Requests
 
@@ -65,6 +55,12 @@ You can also create your custom Import and ImportedObject classes if you need th
   (if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)
 * Send me a pull request. Bonus points for topic branches.
 
+== Disclaimer
+
+Importer gem/plugin was extracted from an actual Ruby on Rails application. It probably lacks some features or needs some polishing. Feel free to contribute.
+
+The gem is still under development, backward compatibility can not be guaranteed (at least until it reaches 1.0 stable version).
+
 == Copyright
 
 Copyright (c) 2010 Michal Szajbe. See LICENSE for details.

data/Rakefile

@@ -5,18 +5,16 @@ begin
   require 'jeweler'
   Jeweler::Tasks.new do |gem|
     gem.name = "importer"
-    gem.summary = %Q{Import objects from XML files}
-    gem.description = %Q{Define new objects or modifications of existing ones in XML file and import them to your application.}
+    gem.summary = %Q{Import objects from external files}
+    gem.description = %Q{Define new objects or modifications of existing ones in external file (xml, csv, etc) and import them to your application. Importer will not only import all the objects but also will give you detailed summary of the import process.}
     gem.email = "michal.szajbe@gmail.com"
     gem.homepage = "http://github.com/szajbus/importer"
     gem.authors = ["Michał Szajbe"]
-    gem.add_dependency "crack", ">= 0"
-    gem.add_dependency "fastercsv", ">= 0"
-    gem.add_dependency "activerecord", ">= 0"
-    gem.add_dependency "activesupport", ">= 0"
-    gem.add_dependency "workflow", ">= 0"
+    gem.add_dependency "crack", ">= 0.1.6"
+    gem.add_dependency "fastercsv", ">= 1.5.0"
+    gem.add_development_dependency "activerecord", ">= 0"
+    gem.add_development_dependency "mongo_mapper", ">= 0.7.0"
     gem.add_development_dependency "thoughtbot-shoulda", ">= 0"
-    gem.add_development_dependency "factory_girl", ">= 0"
     gem.add_development_dependency 'sqlite3-ruby'
     # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
   end

data/VERSION

@@ -1 +1 @@
-0.3.2
+0.4.0

data/importer.gemspec

@@ -5,12 +5,12 @@
 
 Gem::Specification.new do |s|
   s.name = %q{importer}
-  s.version = "0.3.2"
+  s.version = "0.4.0"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["Micha\305\202 Szajbe"]
-  s.date = %q{2010-03-25}
-  s.description = %q{Define new objects or modifications of existing ones in XML file and import them to your application.}
+  s.date = %q{2010-04-07}
+  s.description = %q{Define new objects or modifications of existing ones in external file (xml, csv, etc) and import them to your application. Importer will not only import all the objects but also will give you detailed summary of the import process.}
   s.email = %q{michal.szajbe@gmail.com}
   s.extra_rdoc_files = [
     "LICENSE",
@@ -25,20 +25,15 @@ Gem::Specification.new do |s|
      "VERSION",
      "importer.gemspec",
      "lib/importer.rb",
+     "lib/importer/adapters/active_record_adapter.rb",
+     "lib/importer/adapters/mongo_mapper_adapter.rb",
      "lib/importer/import.rb",
-     "lib/importer/import/active_record.rb",
-     "lib/importer/import/simple.rb",
      "lib/importer/imported_object.rb",
-     "lib/importer/imported_object/active_record.rb",
-     "lib/importer/imported_object/simple.rb",
      "lib/importer/parser.rb",
      "lib/importer/parser/base.rb",
      "lib/importer/parser/csv.rb",
      "lib/importer/parser/xml.rb",
      "rails/init.rb",
-     "rails_generators/importer/importer_generator.rb",
-     "rails_generators/importer/templates/imported_objects_migration.rb",
-     "rails_generators/importer/templates/imports_migration.rb",
      "test/database.yml",
      "test/factories.rb",
      "test/fixtures/empty.csv",
@@ -48,31 +43,29 @@ Gem::Specification.new do |s|
      "test/fixtures/products.csv",
      "test/fixtures/products.xml",
      "test/helper.rb",
-     "test/importer/import/active_record_test.rb",
-     "test/importer/import/simple_test.rb",
-     "test/importer/imported_object/active_record_test.rb",
-     "test/importer/imported_object/simple_test.rb",
+     "test/importer/adapters/active_record_adapter_test.rb",
+     "test/importer/adapters/mongo_mapper_adapter_test.rb",
+     "test/importer/import_test.rb",
+     "test/importer/imported_object_test.rb",
      "test/importer/parser/csv_test.rb",
      "test/importer/parser/xml_test.rb",
-     "test/importer/parser_test.rb",
-     "test/importer_test.rb"
+     "test/importer/parser_test.rb"
   ]
   s.homepage = %q{http://github.com/szajbus/importer}
   s.rdoc_options = ["--charset=UTF-8"]
   s.require_paths = ["lib"]
   s.rubygems_version = %q{1.3.5}
-  s.summary = %q{Import objects from XML files}
+  s.summary = %q{Import objects from external files}
   s.test_files = [
     "test/factories.rb",
      "test/helper.rb",
-     "test/importer/import/active_record_test.rb",
-     "test/importer/import/simple_test.rb",
-     "test/importer/imported_object/active_record_test.rb",
-     "test/importer/imported_object/simple_test.rb",
+     "test/importer/adapters/active_record_adapter_test.rb",
+     "test/importer/adapters/mongo_mapper_adapter_test.rb",
+     "test/importer/import_test.rb",
+     "test/importer/imported_object_test.rb",
      "test/importer/parser/csv_test.rb",
      "test/importer/parser/xml_test.rb",
-     "test/importer/parser_test.rb",
-     "test/importer_test.rb"
+     "test/importer/parser_test.rb"
   ]
 
   if s.respond_to? :specification_version then
@@ -80,32 +73,26 @@ Gem::Specification.new do |s|
     s.specification_version = 3
 
     if Gem::Version.new(Gem::RubyGemsVersion) >= Gem::Version.new('1.2.0') then
-      s.add_runtime_dependency(%q<crack>, [">= 0"])
-      s.add_runtime_dependency(%q<fastercsv>, [">= 0"])
-      s.add_runtime_dependency(%q<activerecord>, [">= 0"])
-      s.add_runtime_dependency(%q<activesupport>, [">= 0"])
-      s.add_runtime_dependency(%q<workflow>, [">= 0"])
+      s.add_runtime_dependency(%q<crack>, [">= 0.1.6"])
+      s.add_runtime_dependency(%q<fastercsv>, [">= 1.5.0"])
+      s.add_development_dependency(%q<activerecord>, [">= 0"])
+      s.add_development_dependency(%q<mongo_mapper>, [">= 0.7.0"])
       s.add_development_dependency(%q<thoughtbot-shoulda>, [">= 0"])
-      s.add_development_dependency(%q<factory_girl>, [">= 0"])
       s.add_development_dependency(%q<sqlite3-ruby>, [">= 0"])
     else
-      s.add_dependency(%q<crack>, [">= 0"])
-      s.add_dependency(%q<fastercsv>, [">= 0"])
+      s.add_dependency(%q<crack>, [">= 0.1.6"])
+      s.add_dependency(%q<fastercsv>, [">= 1.5.0"])
       s.add_dependency(%q<activerecord>, [">= 0"])
-      s.add_dependency(%q<activesupport>, [">= 0"])
-      s.add_dependency(%q<workflow>, [">= 0"])
+      s.add_dependency(%q<mongo_mapper>, [">= 0.7.0"])
       s.add_dependency(%q<thoughtbot-shoulda>, [">= 0"])
-      s.add_dependency(%q<factory_girl>, [">= 0"])
       s.add_dependency(%q<sqlite3-ruby>, [">= 0"])
     end
   else
-    s.add_dependency(%q<crack>, [">= 0"])
-    s.add_dependency(%q<fastercsv>, [">= 0"])
+    s.add_dependency(%q<crack>, [">= 0.1.6"])
+    s.add_dependency(%q<fastercsv>, [">= 1.5.0"])
     s.add_dependency(%q<activerecord>, [">= 0"])
-    s.add_dependency(%q<activesupport>, [">= 0"])
-    s.add_dependency(%q<workflow>, [">= 0"])
+    s.add_dependency(%q<mongo_mapper>, [">= 0.7.0"])
     s.add_dependency(%q<thoughtbot-shoulda>, [">= 0"])
-    s.add_dependency(%q<factory_girl>, [">= 0"])
     s.add_dependency(%q<sqlite3-ruby>, [">= 0"])
   end
 end

data/lib/importer.rb

@@ -1,17 +1,14 @@
-require 'active_support'
-
+require 'importer/adapters/active_record_adapter'
+require 'importer/adapters/mongo_mapper_adapter'
 require 'importer/import'
 require 'importer/imported_object'
-require 'importer/import/active_record'
-require 'importer/import/simple'
-require 'importer/imported_object/active_record'
-require 'importer/imported_object/simple'
 require 'importer/parser'
 require 'importer/parser/base'
 require 'importer/parser/csv'
 require 'importer/parser/xml'
 
-# Importer is a tool to help you with importing objects to your database from various sources.
+# Importer module provides your models with flexible API that makes it easier
+# to import data from external sources.
 #
 # Usage:
 #
@@ -27,74 +24,18 @@ require 'importer/parser/xml'
 # Copyright:: Copyright (c) 2010 Michal Szajbe
 # License:: check the LICENCE file
 module Importer
+  class AdapterError < ::Exception; end
 
   class << self
     def included(base)
-      base.send(:include, Importer::InstanceMethods)
-      base.send(:extend,  Importer::ClassMethods)
-    end
-  end
-
-  module ClassMethods
-    # The import process is wrapped in a transaction, so if anything goes wrong there is no
-    # harm done.
-    # * +file+ - path to an XML or CVS file, you can also import from other data formats,
-    #   but you also need to provide a custom parser to read it
-    # Possible options:
-    # * +parser+ - by default the parser is determined from file extension, but you can force
-    #   the imported to use another one by passing it's class here
-    # * +import+ - by default importer tries to store import summary in database, so it uses
-    #   ActiveRecord import, to use other import type pass it's instance here
-    def import(file, options = {})
-      import = options[:import] || Importer::Import::ActiveRecord.create
-      parser = options[:parser] || Importer::Parser.get_klass(file)
-      data   = parser.run(file)
-
-      transaction do
-        import.start!
-
-        data.each do |attributes|
-          imported_object = import.build_imported_object
-
-          if object = find_on_import(import, attributes)
-            imported_object.state = "existing_object"
-          else
-            object                = new
-            imported_object.state = "new_object"
-          end
-
-          imported_object.data = attributes
-          object.merge_attributes_on_import(import, attributes)
-
-          unless object.save
-            imported_object.state             = "invalid_object"
-            imported_object.validation_errors = object.errors.full_messages
-          end
-
-          imported_object.object = object
-          imported_object.save
-        end
-
-        import.finish!
+      if base.respond_to?(:descends_from_active_record?) && base.descends_from_active_record?
+        base.send(:include, Importer::Adapters::ActiveRecordAdapter)
+      elsif defined?(MongoMapper) && (base.include?(MongoMapper::Document) || base.include?(MongoMapper::EmbeddedDocument))
+        base.send(:include, Importer::Adapters::MongoMapperAdapter)
+      else
+        raise AdapterError.new("Can't determine adapter for #{base.class} class.")
       end
-
-      import
-    end
-
-    # Overload +find_on_import+ method to find existing objects while importing an object.
-    # This is used to determine wheter an imported should add a new object or just modify
-    # an existing one. By default it searches records by id.
-    def find_on_import(import, attributes)
-      find_by_id(attributes["id"])
     end
   end
 
-  module InstanceMethods
-    # Overload +merge_attributes_on_import+ method if you need the detected attributes
-    # merged to an object in some specific way. By default detected attributes are literally
-    # assigned to an object.
-    def merge_attributes_on_import(import, attributes)
-      self.attributes = attributes
-    end
-  end
 end

data/lib/importer/adapters/active_record_adapter.rb

@@ -0,0 +1,102 @@
+module Importer
+  module Adapters
+    # Adapter for ActiveRecord models
+    #
+    # Usage:
+    #
+    #   class Product < ActiveRecord::Base
+    #     include Importer
+    #   end
+    #
+    #   Product.import(path_to_xml_or_csv_file)
+    #
+    # It sends the given file to a parser and then imports detected objects.
+    # Instead of simply inserting all detected objects to database, the importer
+    # tries to determine wheter a detected object already exists. If so, the object
+    # is only updated, otherwise a new object is created.
+    #
+    # To change the way how importer checks for existing objects (or to turn off this
+    # behavior completely) override +find_on_import+ method. The default behavior now
+    # is to try to find existing object by detected object's id.
+    #
+    # By default the detected object's attributes hash is literally assigned to a
+    # soon-to-be-saved object. If there is a need for more sophisticated behavior,
+    # simply override +merge_attributes_on_import+ method.
+    module ActiveRecordAdapter
+
+      class << self
+        def included(base)
+          base.send(:include, InstanceMethods)
+          base.send(:extend,  ClassMethods)
+        end
+      end
+
+      module ClassMethods
+        # The import process is wrapped in a transaction, so if anything goes wrong there is no
+        # harm done.
+        # * +file+ - path to an XML or CVS file, you can also import from other data formats,
+        #   but you also need to provide a custom parser to read it
+        # Possible options:
+        # * +parser+ - by default the parser is determined from file extension, but you can force
+        #   the imported to use another one by passing it's class here
+        # * +import+ - by default importer returns instance of +Import+ class that contains
+        #   detailed report of import process, you can implement your own Import class and force
+        #   the importer to use it by passing it's class here
+        def import(file, options = {})
+          import = (options[:import] || Importer::Import).new
+          parser =  options[:parser] || Importer::Parser.get_klass(file)
+          data   = parser.run(file)
+
+          transaction do
+            data.each do |attributes|
+              imported_object = import.build_imported_object
+
+              if object = find_on_import(import, attributes)
+                imported_object.state = "existing_object"
+              else
+                object                = new
+                imported_object.state = "new_object"
+              end
+
+              imported_object.data = attributes
+              object.merge_attributes_on_import(import, attributes)
+
+              unless object.save
+                imported_object.state             = "invalid_object"
+                imported_object.validation_errors = object.errors.full_messages
+              end
+
+              imported_object.object = object
+
+              import.add_object(imported_object)
+            end
+          end
+
+          import
+        end
+
+        # Determines whether a detected object already exists in database.
+        # By default it tries to find an existing objects by id of the detected one.
+        # Returns the object or nil if it's not found.
+        # Override this method in your model to change that default behavior.
+        # * +import+ - current import
+        # * +attributes+ - detected object's attributes hash
+        def find_on_import(import, attributes)
+          find_by_id(attributes["id"])
+        end
+      end
+
+      module InstanceMethods
+        # Merges attributes of a detected object with current object's ones.
+        # By default it simply assigns detected attributes to the object.
+        # Override this method in your model to provide some more sophisticated
+        # behavior.
+        # * +import+ - current import
+        # * +attributes+ - detected object's attributes hash
+        def merge_attributes_on_import(import, attributes)
+          self.attributes = attributes
+        end
+      end
+    end
+  end
+end

data/lib/importer/adapters/mongo_mapper_adapter.rb

@@ -0,0 +1,104 @@
+module Importer
+  module Adapters
+    # Adapter for MongoMapper models
+    #
+    # Usage:
+    #
+    #   class Product
+    #     include MongoMapper::Document
+    #     include Importer
+    #   end
+    #
+    #   Product.import(path_to_xml_or_csv_file)
+    #
+    # It sends the given file to a parser and then imports detected objects.
+    # Instead of simply inserting all detected objects to database, the importer
+    # tries to determine wheter a detected object already exists. If so, the object
+    # is only updated, otherwise a new object is created.
+    #
+    # To change the way how importer checks for existing objects (or to turn off this
+    # behavior completely) override +find_on_import+ method. The default behavior now
+    # is to try to find existing object by detected object's id.
+    #
+    # By default the detected object's attributes hash is literally assigned to a
+    # soon-to-be-saved object. If there is a need for more sophisticated behavior,
+    # simply override +merge_attributes_on_import+ method.
+    module MongoMapperAdapter
+
+      class << self
+        def included(base)
+          base.send(:include, InstanceMethods)
+          base.send(:extend,  ClassMethods)
+        end
+      end
+
+      module ClassMethods
+        # Performs actual import
+        #
+        # Note: unlike with ActiveRecord adapter, import process is not wrapped in a transaction
+        # since mongodb does not support them.
+        #
+        # * +file+ - path to an XML or CVS file, you can also import from other data formats,
+        #   but you also need to provide a custom parser to read it
+        # Possible options:
+        # * +parser+ - by default the parser is determined from file extension, but you can force
+        #   the imported to use another one by passing it's class here
+        # * +import+ - by default importer returns instance of +Import+ class that contains
+        #   detailed report of import process, you can implement your own Import class and force
+        #   the importer to use it by passing it's class here
+        def import(file, options = {})
+          import = (options[:import] || Importer::Import).new
+          parser =  options[:parser] || Importer::Parser.get_klass(file)
+          data   = parser.run(file)
+
+          data.each do |attributes|
+            imported_object = import.build_imported_object
+
+            if object = find_on_import(import, attributes)
+              imported_object.state = "existing_object"
+            else
+              object                = new
+              imported_object.state = "new_object"
+            end
+
+            imported_object.data = attributes
+            object.merge_attributes_on_import(import, attributes)
+
+            unless object.save
+              imported_object.state             = "invalid_object"
+              imported_object.validation_errors = object.errors.full_messages
+            end
+
+            imported_object.object = object
+
+            import.add_object(imported_object)
+          end
+
+          import
+        end
+
+        # Determines whether a detected object already exists in database.
+        # By default it tries to find an existing objects by id of the detected one.
+        # Returns the object or nil if it's not found.
+        # Override this method in your model to change that default behavior.
+        # * +import+ - current import
+        # * +attributes+ - detected object's attributes hash
+        def find_on_import(import, attributes)
+          find_by_id(attributes["id"])
+        end
+      end
+
+      module InstanceMethods
+        # Merges attributes of a detected object with current object's ones.
+        # By default it simply assigns detected attributes to the object.
+        # Override this method in your model to provide some more sophisticated
+        # behavior.
+        # * +import+ - current import
+        # * +attributes+ - detected object's attributes hash
+        def merge_attributes_on_import(import, attributes)
+          self.attributes = attributes
+        end
+      end
+    end
+  end
+end