importer 0.3.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (41) hide show
  1. data/.document +5 -0
  2. data/.gitignore +21 -0
  3. data/LICENSE +20 -0
  4. data/README.rdoc +70 -0
  5. data/Rakefile +60 -0
  6. data/VERSION +1 -0
  7. data/importer.gemspec +114 -0
  8. data/lib/importer.rb +102 -0
  9. data/lib/importer/import.rb +5 -0
  10. data/lib/importer/import/active_record.rb +37 -0
  11. data/lib/importer/import/simple.rb +57 -0
  12. data/lib/importer/imported_object.rb +7 -0
  13. data/lib/importer/imported_object/active_record.rb +41 -0
  14. data/lib/importer/imported_object/simple.rb +25 -0
  15. data/lib/importer/parser.rb +22 -0
  16. data/lib/importer/parser/base.rb +31 -0
  17. data/lib/importer/parser/csv.rb +24 -0
  18. data/lib/importer/parser/xml.rb +25 -0
  19. data/rails/init.rb +1 -0
  20. data/rails_generators/importer/importer_generator.rb +18 -0
  21. data/rails_generators/importer/templates/imported_objects_migration.rb +19 -0
  22. data/rails_generators/importer/templates/imports_migration.rb +17 -0
  23. data/test/database.yml +3 -0
  24. data/test/factories.rb +11 -0
  25. data/test/fixtures/empty.csv +0 -0
  26. data/test/fixtures/empty.xml +3 -0
  27. data/test/fixtures/product.csv +2 -0
  28. data/test/fixtures/product.xml +9 -0
  29. data/test/fixtures/products.csv +4 -0
  30. data/test/fixtures/products.xml +21 -0
  31. data/test/helper.rb +71 -0
  32. data/test/importer/import/active_record_test.rb +27 -0
  33. data/test/importer/import/simple_test.rb +27 -0
  34. data/test/importer/imported_object/active_record_test.rb +37 -0
  35. data/test/importer/imported_object/simple_test.rb +35 -0
  36. data/test/importer/imported_object_test.rb +13 -0
  37. data/test/importer/parser/csv_test.rb +59 -0
  38. data/test/importer/parser/xml_test.rb +58 -0
  39. data/test/importer/parser_test.rb +13 -0
  40. data/test/importer_test.rb +50 -0
  41. metadata +184 -0
@@ -0,0 +1,5 @@
1
+ README.rdoc
2
+ lib/**/*.rb
3
+ bin/*
4
+ features/**/*.feature
5
+ LICENSE
@@ -0,0 +1,21 @@
1
+ ## MAC OS
2
+ .DS_Store
3
+
4
+ ## TEXTMATE
5
+ *.tmproj
6
+ tmtags
7
+
8
+ ## EMACS
9
+ *~
10
+ \#*
11
+ .\#*
12
+
13
+ ## VIM
14
+ *.swp
15
+
16
+ ## PROJECT::GENERAL
17
+ coverage
18
+ rdoc
19
+ pkg
20
+
21
+ ## PROJECT::SPECIFIC
data/LICENSE ADDED
@@ -0,0 +1,20 @@
1
+ Copyright (c) 2010 Michał Szajbe
2
+
3
+ Permission is hereby granted, free of charge, to any person obtaining
4
+ a copy of this software and associated documentation files (the
5
+ "Software"), to deal in the Software without restriction, including
6
+ without limitation the rights to use, copy, modify, merge, publish,
7
+ distribute, sublicense, and/or sell copies of the Software, and to
8
+ permit persons to whom the Software is furnished to do so, subject to
9
+ the following conditions:
10
+
11
+ The above copyright notice and this permission notice shall be
12
+ included in all copies or substantial portions of the Software.
13
+
14
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
15
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
16
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
17
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
18
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
19
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
20
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,70 @@
1
+ = importer
2
+
3
+ Importer is a tool to help you with importing objects to your database from various sources.
4
+
5
+ === Links
6
+
7
+ Repository: http://github.com/szajbus/importer
8
+
9
+ Docs: http://rdoc.info/projects/szajbus/importer
10
+
11
+ == Main features:
12
+
13
+ * can import from XML and CSV formats, but it's possible to add custom parsers
14
+ * reports how many new objects got imported, how many objects was modified and how many objects were invalid
15
+ * can save reports to the database, along with errors for later inspection
16
+
17
+ == Installation
18
+
19
+ Install the gem
20
+
21
+ gem install importer
22
+
23
+ Or the plugin
24
+
25
+ script/plugin install git://github.com/szajbus/importer.git
26
+
27
+ Generate the migrations and run them (this step may not be necessary, read below)
28
+
29
+ script/generate importer import
30
+ rake db:migrate
31
+
32
+ Add to your model
33
+
34
+ class Product < ActiveRecord::Base
35
+ include Importer
36
+ end
37
+
38
+ And start importing
39
+
40
+ Product.import(path_to_xml_or_csv_file)
41
+
42
+ This will parse the file and import all products there are defined in it. Import summary will be saved in imports table, it will tell you how many products were created, modified or invalid. Exact information about each product (detected attributes and errors) will be saved in imported_objects table.
43
+
44
+ If you don't want to save summaries to database you can force the importer to do a simple import with:
45
+
46
+ Product.import(path_to_xml_or_csv_file, :import => Importer::Import::Simple.create)
47
+
48
+ == Customization
49
+
50
+ You can create your own parser to import from sources other than XML or CSV files. Check the implementation of one of existing parsers to find out how to write your own. Then just pass parser class to import method:
51
+
52
+ Product.import(path_to_xml_or_csv_file, :parser => YourCustomParser)
53
+
54
+ You can also create your custom Import and ImportedObject classes if you need the import summaries in other way than those currently offered (ActiveRecord which stores summaries in database and Simple which does not store them at all). Again check the current implementations to get the idea. To force the importer to use it:
55
+
56
+ Product.import(path_to_xml_or_csv_file, :import => YourCustomImport.new)
57
+
58
+ == Note on Patches/Pull Requests
59
+
60
+ * Fork the project.
61
+ * Make your feature addition or bug fix.
62
+ * Add tests for it. This is important so I don't break it in a
63
+ future version unintentionally.
64
+ * Commit, do not mess with rakefile, version, or history.
65
+ (if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)
66
+ * Send me a pull request. Bonus points for topic branches.
67
+
68
+ == Copyright
69
+
70
+ Copyright (c) 2010 Michal Szajbe. See LICENSE for details.
@@ -0,0 +1,60 @@
1
+ require 'rubygems'
2
+ require 'rake'
3
+
4
+ begin
5
+ require 'jeweler'
6
+ Jeweler::Tasks.new do |gem|
7
+ gem.name = "importer"
8
+ gem.summary = %Q{Import objects from XML files}
9
+ gem.description = %Q{Define new objects or modifications of existing ones in XML file and import them to your application.}
10
+ gem.email = "michal.szajbe@gmail.com"
11
+ gem.homepage = "http://github.com/szajbus/importer"
12
+ gem.authors = ["Michał Szajbe"]
13
+ gem.add_dependency "crack", ">= 0"
14
+ gem.add_dependency "fastercsv", ">= 0"
15
+ gem.add_dependency "activerecord", ">= 0"
16
+ gem.add_dependency "activesupport", ">= 0"
17
+ gem.add_dependency "workflow", ">= 0"
18
+ gem.add_development_dependency "thoughtbot-shoulda", ">= 0"
19
+ gem.add_development_dependency "factory_girl", ">= 0"
20
+ gem.add_development_dependency 'sqlite3-ruby'
21
+ # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
22
+ end
23
+ Jeweler::GemcutterTasks.new
24
+ rescue LoadError
25
+ puts "Jeweler (or a dependency) not available. Install it with: gem install jeweler"
26
+ end
27
+
28
+ require 'rake/testtask'
29
+ Rake::TestTask.new(:test) do |test|
30
+ test.libs << 'lib' << 'test'
31
+ test.pattern = 'test/**/*_test.rb'
32
+ test.verbose = true
33
+ end
34
+
35
+ begin
36
+ require 'rcov/rcovtask'
37
+ Rcov::RcovTask.new do |test|
38
+ test.libs << 'test'
39
+ test.pattern = 'test/**/*_test.rb'
40
+ test.verbose = true
41
+ end
42
+ rescue LoadError
43
+ task :rcov do
44
+ abort "RCov is not available. In order to run rcov, you must: sudo gem install spicycode-rcov"
45
+ end
46
+ end
47
+
48
+ task :test => :check_dependencies
49
+
50
+ task :default => :test
51
+
52
+ require 'rake/rdoctask'
53
+ Rake::RDocTask.new do |rdoc|
54
+ version = File.exist?('VERSION') ? File.read('VERSION') : ""
55
+
56
+ rdoc.rdoc_dir = 'rdoc'
57
+ rdoc.title = "importer #{version}"
58
+ rdoc.rdoc_files.include('README*')
59
+ rdoc.rdoc_files.include('lib/**/*.rb')
60
+ end
data/VERSION ADDED
@@ -0,0 +1 @@
1
+ 0.3.0
@@ -0,0 +1,114 @@
1
+ # Generated by jeweler
2
+ # DO NOT EDIT THIS FILE DIRECTLY
3
+ # Instead, edit Jeweler::Tasks in Rakefile, and run the gemspec command
4
+ # -*- encoding: utf-8 -*-
5
+
6
+ Gem::Specification.new do |s|
7
+ s.name = %q{importer}
8
+ s.version = "0.3.0"
9
+
10
+ s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
11
+ s.authors = ["Micha\305\202 Szajbe"]
12
+ s.date = %q{2010-03-20}
13
+ s.description = %q{Define new objects or modifications of existing ones in XML file and import them to your application.}
14
+ s.email = %q{michal.szajbe@gmail.com}
15
+ s.extra_rdoc_files = [
16
+ "LICENSE",
17
+ "README.rdoc"
18
+ ]
19
+ s.files = [
20
+ ".document",
21
+ ".gitignore",
22
+ "LICENSE",
23
+ "README.rdoc",
24
+ "Rakefile",
25
+ "VERSION",
26
+ "importer.gemspec",
27
+ "lib/importer.rb",
28
+ "lib/importer/import.rb",
29
+ "lib/importer/import/active_record.rb",
30
+ "lib/importer/import/simple.rb",
31
+ "lib/importer/imported_object.rb",
32
+ "lib/importer/imported_object/active_record.rb",
33
+ "lib/importer/imported_object/simple.rb",
34
+ "lib/importer/parser.rb",
35
+ "lib/importer/parser/base.rb",
36
+ "lib/importer/parser/csv.rb",
37
+ "lib/importer/parser/xml.rb",
38
+ "rails/init.rb",
39
+ "rails_generators/importer/importer_generator.rb",
40
+ "rails_generators/importer/templates/imported_objects_migration.rb",
41
+ "rails_generators/importer/templates/imports_migration.rb",
42
+ "test/database.yml",
43
+ "test/factories.rb",
44
+ "test/fixtures/empty.csv",
45
+ "test/fixtures/empty.xml",
46
+ "test/fixtures/product.csv",
47
+ "test/fixtures/product.xml",
48
+ "test/fixtures/products.csv",
49
+ "test/fixtures/products.xml",
50
+ "test/helper.rb",
51
+ "test/importer/import/active_record_test.rb",
52
+ "test/importer/import/simple_test.rb",
53
+ "test/importer/imported_object/active_record_test.rb",
54
+ "test/importer/imported_object/simple_test.rb",
55
+ "test/importer/imported_object_test.rb",
56
+ "test/importer/parser/csv_test.rb",
57
+ "test/importer/parser/xml_test.rb",
58
+ "test/importer/parser_test.rb",
59
+ "test/importer_test.rb"
60
+ ]
61
+ s.homepage = %q{http://github.com/szajbus/importer}
62
+ s.rdoc_options = ["--charset=UTF-8"]
63
+ s.require_paths = ["lib"]
64
+ s.rubygems_version = %q{1.3.5}
65
+ s.summary = %q{Import objects from XML files}
66
+ s.test_files = [
67
+ "test/factories.rb",
68
+ "test/helper.rb",
69
+ "test/importer/import/active_record_test.rb",
70
+ "test/importer/import/simple_test.rb",
71
+ "test/importer/imported_object/active_record_test.rb",
72
+ "test/importer/imported_object/simple_test.rb",
73
+ "test/importer/imported_object_test.rb",
74
+ "test/importer/parser/csv_test.rb",
75
+ "test/importer/parser/xml_test.rb",
76
+ "test/importer/parser_test.rb",
77
+ "test/importer_test.rb"
78
+ ]
79
+
80
+ if s.respond_to? :specification_version then
81
+ current_version = Gem::Specification::CURRENT_SPECIFICATION_VERSION
82
+ s.specification_version = 3
83
+
84
+ if Gem::Version.new(Gem::RubyGemsVersion) >= Gem::Version.new('1.2.0') then
85
+ s.add_runtime_dependency(%q<crack>, [">= 0"])
86
+ s.add_runtime_dependency(%q<fastercsv>, [">= 0"])
87
+ s.add_runtime_dependency(%q<activerecord>, [">= 0"])
88
+ s.add_runtime_dependency(%q<activesupport>, [">= 0"])
89
+ s.add_runtime_dependency(%q<workflow>, [">= 0"])
90
+ s.add_development_dependency(%q<thoughtbot-shoulda>, [">= 0"])
91
+ s.add_development_dependency(%q<factory_girl>, [">= 0"])
92
+ s.add_development_dependency(%q<sqlite3-ruby>, [">= 0"])
93
+ else
94
+ s.add_dependency(%q<crack>, [">= 0"])
95
+ s.add_dependency(%q<fastercsv>, [">= 0"])
96
+ s.add_dependency(%q<activerecord>, [">= 0"])
97
+ s.add_dependency(%q<activesupport>, [">= 0"])
98
+ s.add_dependency(%q<workflow>, [">= 0"])
99
+ s.add_dependency(%q<thoughtbot-shoulda>, [">= 0"])
100
+ s.add_dependency(%q<factory_girl>, [">= 0"])
101
+ s.add_dependency(%q<sqlite3-ruby>, [">= 0"])
102
+ end
103
+ else
104
+ s.add_dependency(%q<crack>, [">= 0"])
105
+ s.add_dependency(%q<fastercsv>, [">= 0"])
106
+ s.add_dependency(%q<activerecord>, [">= 0"])
107
+ s.add_dependency(%q<activesupport>, [">= 0"])
108
+ s.add_dependency(%q<workflow>, [">= 0"])
109
+ s.add_dependency(%q<thoughtbot-shoulda>, [">= 0"])
110
+ s.add_dependency(%q<factory_girl>, [">= 0"])
111
+ s.add_dependency(%q<sqlite3-ruby>, [">= 0"])
112
+ end
113
+ end
114
+
@@ -0,0 +1,102 @@
1
+ require 'active_support'
2
+
3
+ require 'importer/import'
4
+ require 'importer/imported_object'
5
+ require 'importer/import/active_record'
6
+ require 'importer/import/simple'
7
+ require 'importer/imported_object/active_record'
8
+ require 'importer/imported_object/simple'
9
+ require 'importer/parser'
10
+ require 'importer/parser/base'
11
+ require 'importer/parser/csv'
12
+ require 'importer/parser/xml'
13
+
14
+ # Importer is a tool to help you with importing objects to your database from various sources.
15
+ #
16
+ # Usage:
17
+ #
18
+ # class Product < ActiveRecord::Base
19
+ # include Importer
20
+ # end
21
+ #
22
+ # Product.import(path_to_xml_or_csv_file)
23
+ #
24
+ # Check README.rdoc file for more useful information
25
+ #
26
+ # Author:: Michal Szajbe
27
+ # Copyright:: Copyright (c) 2010 Michal Szajbe
28
+ # License:: check the LICENCE file
29
+ module Importer
30
+
31
+ class << self
32
+ def included(base)
33
+ base.send(:include, Importer::InstanceMethods)
34
+ base.send(:extend, Importer::ClassMethods)
35
+ end
36
+ end
37
+
38
+ module ClassMethods
39
+ # The import process is wrapped in a transaction, so if anything goes wrong there is no
40
+ # harm done.
41
+ # * +file+ - path to an XML or CVS file, you can also import from other data formats,
42
+ # but you also need to provide a custom parser to read it
43
+ # Possible options:
44
+ # * +parser+ - by default the parser is determined from file extension, but you can force
45
+ # the imported to use another one by passing it's class here
46
+ # * +import+ - by default importer tries to store import summary in database, so it uses
47
+ # ActiveRecord import, to use other import type pass it's instance here
48
+ def import(file, options = {})
49
+ import = options[:import] || Importer::Import::ActiveRecord.create
50
+ parser = options[:parser] || Importer::Parser.get_klass(file)
51
+ data = parser.run(file)
52
+
53
+ transaction do
54
+ import.start!
55
+
56
+ imported_object_klass = Importer::ImportedObject.get_klass(import)
57
+
58
+ data.each do |attributes|
59
+ imported_object = imported_object_klass.new(:import => import)
60
+
61
+ if object = find_on_import(import, attributes)
62
+ imported_object.state = "existing_object"
63
+ else
64
+ object = new
65
+ imported_object.state = "new_object"
66
+ end
67
+
68
+ imported_object.data = attributes
69
+ object.merge_attributes_on_import(import, attributes)
70
+
71
+ unless object.save
72
+ imported_object.state = "invalid_object"
73
+ imported_object.validation_errors = object.errors.full_messages
74
+ end
75
+
76
+ imported_object.object = object
77
+ imported_object.save
78
+ end
79
+
80
+ import.finish!
81
+ end
82
+
83
+ import
84
+ end
85
+
86
+ # Overload +find_on_import+ method to find existing objects while importing an object.
87
+ # This is used to determine wheter an imported should add a new object or just modify
88
+ # an existing one. By default it searches records by id.
89
+ def find_on_import(import, attributes)
90
+ find_by_id(attributes["id"])
91
+ end
92
+ end
93
+
94
+ module InstanceMethods
95
+ # Overload +merge_attributes_on_import+ method if you need the detected attributes
96
+ # merged to an object in some specific way. By default detected attributes are literally
97
+ # assigned to an object.
98
+ def merge_attributes_on_import(import, attributes)
99
+ self.attributes = attributes
100
+ end
101
+ end
102
+ end
@@ -0,0 +1,5 @@
1
+ module Importer
2
+ module Import
3
+
4
+ end
5
+ end
@@ -0,0 +1,37 @@
1
+ require 'active_record'
2
+ require 'workflow'
3
+
4
+ module Importer
5
+ module Import
6
+ # ActiveRecord model that stores import summaries in imports database table.
7
+ # has_many :+imported_objects+
8
+ #
9
+ # Attributes:
10
+ # * +new_objects_count+ - number of new objects created during the import
11
+ # * +existing_objects_count+ - number of objects modified during the import
12
+ # * +invalid_objects_count+ - number of objects that couldn't have been imported
13
+ # * +workflow_state+ - import may be in one of three states: ready, started or
14
+ # finished. The state changes during the import process.
15
+ class ActiveRecord < ::ActiveRecord::Base
16
+ set_table_name "imports"
17
+
18
+ include Workflow
19
+
20
+ workflow do
21
+ state :ready do
22
+ event :start, :transitions_to => :started
23
+ end
24
+ state :started do
25
+ event :finish, :transitions_to => :finished
26
+ end
27
+ state :finished
28
+ end
29
+
30
+ named_scope :ready, :conditions => { :workflow_state => "ready" }
31
+ named_scope :started, :conditions => { :workflow_state => "started" }
32
+ named_scope :finished, :conditions => { :workflow_state => "finished" }
33
+
34
+ has_many :imported_objects, :class_name => "Importer::ImportedObject::ActiveRecord"
35
+ end
36
+ end
37
+ end