csv_importable 0.1.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/.gitignore +9 -0
- data/.rspec +2 -0
- data/.travis.yml +5 -0
- data/Gemfile +4 -0
- data/README.md +104 -0
- data/Rakefile +6 -0
- data/bin/console +14 -0
- data/bin/setup +8 -0
- data/csv_importable.gemspec +34 -0
- data/lib/csv_importable.rb +10 -0
- data/lib/csv_importable/csv_coercion.rb +40 -0
- data/lib/csv_importable/csv_importer.rb +192 -0
- data/lib/csv_importable/importable.rb +148 -0
- data/lib/csv_importable/row_importer.rb +22 -0
- data/lib/csv_importable/type_parser.rb +77 -0
- data/lib/csv_importable/type_parser/boolean_type_parser.rb +13 -0
- data/lib/csv_importable/type_parser/date_type_parser.rb +23 -0
- data/lib/csv_importable/type_parser/float_type_parser.rb +11 -0
- data/lib/csv_importable/type_parser/integer_type_parser.rb +11 -0
- data/lib/csv_importable/type_parser/percent_type_parser.rb +22 -0
- data/lib/csv_importable/type_parser/select_type_parser.rb +19 -0
- data/lib/csv_importable/type_parser/string_type_parser.rb +11 -0
- data/lib/csv_importable/version.rb +3 -0
- metadata +124 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: 468b8461398702b643ac57db4fe9aa28309e5abe
|
4
|
+
data.tar.gz: b31a2f2189637a5889d27e846239c06231b3df05
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 37fdcd78cc1b9518d89c8be76167e348ae61f60688e021647589ce273965c65c3cbf734a42ec7bfb10d0d1f9b39c06c358abb7f1e7316e8b5b5f170e3d5c6c23
|
7
|
+
data.tar.gz: 8d95e02da61b4823eb94b3b5bd4723052f35b758d869dc950ffe5b4a43e9c91a2b96b68996e3a66bf44dce94a9b3127d6c1e387b396225fb01ad835d2eac3dd0
|
data/.gitignore
ADDED
data/.rspec
ADDED
data/.travis.yml
ADDED
data/Gemfile
ADDED
data/README.md
ADDED
@@ -0,0 +1,104 @@
|
|
1
|
+
# CSV Importable
|
2
|
+
|
3
|
+
Intelligently parse CSVs and display errors to your users.
|
4
|
+
|
5
|
+
## Installation
|
6
|
+
|
7
|
+
Add this line to your application's Gemfile:
|
8
|
+
|
9
|
+
```ruby
|
10
|
+
gem 'csv_importable'
|
11
|
+
```
|
12
|
+
|
13
|
+
And then execute:
|
14
|
+
|
15
|
+
$ bundle
|
16
|
+
|
17
|
+
## Usage
|
18
|
+
|
19
|
+
High level steps:
|
20
|
+
|
21
|
+
1. Create an `Import` model: this model handles and stores the file, status, and results of the import for the user to see.
|
22
|
+
2. Create a `RowImporter` class: this class handles the logic surrounding how one row in the CSV should be added to the database.
|
23
|
+
|
24
|
+
Please note, it is also possible to implement an `Importer` class, which handles the logic surrounding how the entire file is imported. This is not usually needed though.
|
25
|
+
|
26
|
+
|
27
|
+
### Create Import Model
|
28
|
+
|
29
|
+
This model handles and stores the file, status, and results of the import for the user to see. By storing the file and results, we can process the import in the background when the file is too large to process real-time, and then email the user when the import is finished.
|
30
|
+
|
31
|
+
Note: if you're not using Paperclip, you can modify `file` to be a string or some other data type that helps you find the file for the `read_file` method, which is really the only required field as it relates to the uploaded file.
|
32
|
+
|
33
|
+
$ rails g model Import status:string results:text type:string file:attachment should_replace:boolean
|
34
|
+
$ bundle exec rake db:migrate
|
35
|
+
|
36
|
+
Change the Import class to look something like below:
|
37
|
+
|
38
|
+
```ruby
|
39
|
+
class Import < ActiveRecord::Base
|
40
|
+
include CSVImportable::Importable
|
41
|
+
|
42
|
+
def row_importer_class
|
43
|
+
# e.g. UserRowImporter (see next section)
|
44
|
+
end
|
45
|
+
|
46
|
+
def read_file
|
47
|
+
# needs to return StringIO of file
|
48
|
+
# for paperclip, use:
|
49
|
+
# Paperclip.io_adapters.for(file).read
|
50
|
+
end
|
51
|
+
|
52
|
+
def after_async_complete
|
53
|
+
# this is an optional hook for when an async import finishes
|
54
|
+
# e.g. SiteMailer.import_completed(import).deliver_later
|
55
|
+
end
|
56
|
+
|
57
|
+
def save_to_db
|
58
|
+
save
|
59
|
+
end
|
60
|
+
|
61
|
+
def big_file_threshold
|
62
|
+
# max number of rows before processing with a background job
|
63
|
+
# super returns the default of 10
|
64
|
+
super
|
65
|
+
end
|
66
|
+
end
|
67
|
+
```
|
68
|
+
|
69
|
+
### Create RowImporter Class
|
70
|
+
|
71
|
+
this class handles the logic surrounding how one row in the CSV should be added to the database. You need only (1) inherit from `CSVImportable::CSVImporter` and (2) implement the `import_row` method.
|
72
|
+
|
73
|
+
```ruby
|
74
|
+
class UserRowImporter < CSVImportable::CSVImporter
|
75
|
+
def import_row
|
76
|
+
user = User.new
|
77
|
+
user.email = pull_string('email', required: true)
|
78
|
+
user.first_name = pull_string('first_name', required: true)
|
79
|
+
user.last_name = pull_string('last_name', required: true)
|
80
|
+
user.birthdate = pull_date('birthdate') # format: YYYYMMDD
|
81
|
+
user.salary = pull_float('salary')
|
82
|
+
end
|
83
|
+
end
|
84
|
+
```
|
85
|
+
|
86
|
+
#### Parsers
|
87
|
+
|
88
|
+
To assist you in getting data out of your CSV, we've implemented some basic parsers. These parsers will grab the raw data for the particular row/column and attempt to coerce it into the correct type (e.g. take string from CSV and convert to float).
|
89
|
+
|
90
|
+
If the parser fails to coerce the data properly, it will add an error message to the array of errors that your user receives after the import runs. These errors help the user fix the import file in order to try again.
|
91
|
+
|
92
|
+
- pull_string
|
93
|
+
- pull_boolean
|
94
|
+
- pull_date
|
95
|
+
- pull_float
|
96
|
+
- pull_integer
|
97
|
+
- pull_select
|
98
|
+
|
99
|
+
Basic syntax: `pull_string(column_key, args)` where `column_key` is the CSV header string for the column and `args` is a hash with the following defaults: `{ required: false }`
|
100
|
+
|
101
|
+
## Contributing
|
102
|
+
|
103
|
+
Bug reports and pull requests are welcome on GitHub at https://github.com/launchpadlab/csv_importable
|
104
|
+
|
data/Rakefile
ADDED
data/bin/console
ADDED
@@ -0,0 +1,14 @@
|
|
1
|
+
#!/usr/bin/env ruby
|
2
|
+
|
3
|
+
require "bundler/setup"
|
4
|
+
require "csv_importable"
|
5
|
+
|
6
|
+
# You can add fixtures and/or initialization code here to make experimenting
|
7
|
+
# with your gem easier. You can also use a different console, if you like.
|
8
|
+
|
9
|
+
# (If you use this, don't forget to add pry to your Gemfile!)
|
10
|
+
# require "pry"
|
11
|
+
# Pry.start
|
12
|
+
|
13
|
+
require "irb"
|
14
|
+
IRB.start
|
data/bin/setup
ADDED
@@ -0,0 +1,34 @@
|
|
1
|
+
# coding: utf-8
|
2
|
+
lib = File.expand_path('../lib', __FILE__)
|
3
|
+
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
|
4
|
+
require 'csv_importable/version'
|
5
|
+
|
6
|
+
Gem::Specification.new do |spec|
|
7
|
+
spec.name = "csv_importable"
|
8
|
+
spec.version = CsvImportable::VERSION
|
9
|
+
spec.authors = ["Ryan Francis"]
|
10
|
+
spec.email = ["ryan@launchpadlab.com"]
|
11
|
+
|
12
|
+
spec.summary = %q{Import CSV files from users}
|
13
|
+
spec.description = %q{Equipped with intelligent parsing rules and excellent error handling, this gem is all you need for importing CSVs in Ruby.}
|
14
|
+
spec.homepage = "https://github.com/LaunchPadLab/csv_importable"
|
15
|
+
|
16
|
+
# Prevent pushing this gem to RubyGems.org. To allow pushes either set the 'allowed_push_host'
|
17
|
+
# to allow pushing to a single host or delete this section to allow pushing to any host.
|
18
|
+
if spec.respond_to?(:metadata)
|
19
|
+
spec.metadata['allowed_push_host'] = "https://rubygems.org"
|
20
|
+
else
|
21
|
+
raise "RubyGems 2.0 or newer is required to protect against public gem pushes."
|
22
|
+
end
|
23
|
+
|
24
|
+
spec.files = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
|
25
|
+
spec.bindir = "exe"
|
26
|
+
spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
|
27
|
+
spec.require_paths = ["lib"]
|
28
|
+
|
29
|
+
spec.add_development_dependency "bundler", "~> 1.12"
|
30
|
+
spec.add_development_dependency "rake", "~> 10.0"
|
31
|
+
spec.add_development_dependency "rspec", "~> 3.0"
|
32
|
+
|
33
|
+
spec.add_dependency "asyncable"
|
34
|
+
end
|
@@ -0,0 +1,10 @@
|
|
1
|
+
require "csv_importable/version"
|
2
|
+
|
3
|
+
module CsvImportable
|
4
|
+
end
|
5
|
+
|
6
|
+
require_relative './csv_importable/type_parser'
|
7
|
+
require_relative './csv_importable/csv_coercion'
|
8
|
+
require_relative './csv_importable/row_importer'
|
9
|
+
require_relative './csv_importable/csv_importer'
|
10
|
+
require_relative './csv_importable/importable'
|
@@ -0,0 +1,40 @@
|
|
1
|
+
module CSVImportable
|
2
|
+
module CSVCoercion
|
3
|
+
def string_type_class
|
4
|
+
CSVImportable::TypeParser::StringTypeParser
|
5
|
+
end
|
6
|
+
|
7
|
+
def date_type_class
|
8
|
+
CSVImportable::TypeParser::DateTypeParser
|
9
|
+
end
|
10
|
+
|
11
|
+
def boolean_type_class
|
12
|
+
CSVImportable::TypeParser::BooleanTypeParser
|
13
|
+
end
|
14
|
+
|
15
|
+
def integer_type_class
|
16
|
+
CSVImportable::TypeParser::IntegerTypeParser
|
17
|
+
end
|
18
|
+
|
19
|
+
def float_type_class
|
20
|
+
CSVImportable::TypeParser::FloatTypeParser
|
21
|
+
end
|
22
|
+
|
23
|
+
def percent_type_class
|
24
|
+
CSVImportable::TypeParser::PercentTypeParser
|
25
|
+
end
|
26
|
+
|
27
|
+
def select_type_class
|
28
|
+
CSVImportable::TypeParser::SelectTypeParser
|
29
|
+
end
|
30
|
+
|
31
|
+
[:string, :date, :boolean, :integer, :float, :percent, :select].each do |parser_type|
|
32
|
+
define_method("pull_#{parser_type}") do |key, options={}|
|
33
|
+
csv_row = options.fetch(:row, @row)
|
34
|
+
options = options.merge(row: csv_row)
|
35
|
+
parser = send("#{parser_type}_type_class").new(key, options)
|
36
|
+
parser.parse
|
37
|
+
end
|
38
|
+
end
|
39
|
+
end
|
40
|
+
end
|
@@ -0,0 +1,192 @@
|
|
1
|
+
require 'csv'
|
2
|
+
|
3
|
+
module CSVImportable
|
4
|
+
class CSVImporter
|
5
|
+
include CSVImportable::CSVCoercion
|
6
|
+
attr_reader :file_string, :should_replace, :out, :results,
|
7
|
+
:importable_class, :import_obj, :row_importer_class,
|
8
|
+
:big_file_threshold
|
9
|
+
|
10
|
+
module Statuses
|
11
|
+
SUCCESS = :success
|
12
|
+
ERROR = :error
|
13
|
+
end
|
14
|
+
|
15
|
+
def initialize(args = {})
|
16
|
+
@file_string = args[:file_string]
|
17
|
+
import_id = args[:import_id]
|
18
|
+
@importable_class = args[:importable_class]
|
19
|
+
@big_file_threshold = args[:big_file_threshold]
|
20
|
+
@import_obj = importable_class.find(import_id) if import_id
|
21
|
+
# because we can't pass file_string to delayed job
|
22
|
+
@file_string = @import_obj.read_file if @import_obj
|
23
|
+
@should_replace = args.fetch(:should_replace, false)
|
24
|
+
@row_importer_class = args[:row_importer_class]
|
25
|
+
after_init(args)
|
26
|
+
require_args
|
27
|
+
end
|
28
|
+
|
29
|
+
def out
|
30
|
+
@out ||= StringIO.new
|
31
|
+
end
|
32
|
+
|
33
|
+
def import
|
34
|
+
@results = guard_against_errors do
|
35
|
+
destroy_records if should_replace?
|
36
|
+
print(starting_message)
|
37
|
+
before_rows
|
38
|
+
@results = parse_csv_string(file_string) do |row, headers|
|
39
|
+
process_row(row, headers)
|
40
|
+
end
|
41
|
+
after_rows(@results.map { |result| result[:value] })
|
42
|
+
print(finished_message)
|
43
|
+
@results
|
44
|
+
end
|
45
|
+
@results
|
46
|
+
end
|
47
|
+
|
48
|
+
def big_file?
|
49
|
+
parse_csv_string(file_string).count > big_file_threshold
|
50
|
+
end
|
51
|
+
|
52
|
+
def succeeded?
|
53
|
+
results[:status] == Statuses::SUCCESS
|
54
|
+
end
|
55
|
+
|
56
|
+
def number_imported
|
57
|
+
results.fetch(:results).length
|
58
|
+
end
|
59
|
+
|
60
|
+
private
|
61
|
+
|
62
|
+
def after_init(args = {})
|
63
|
+
# hook for subclasses
|
64
|
+
end
|
65
|
+
|
66
|
+
def require_args
|
67
|
+
required_args.each do |required_arg|
|
68
|
+
fail "#{required_arg} is required for #{self.class.name}" unless send(required_arg)
|
69
|
+
end
|
70
|
+
end
|
71
|
+
|
72
|
+
def before_rows
|
73
|
+
# hook for subclasses
|
74
|
+
end
|
75
|
+
|
76
|
+
def after_rows(values)
|
77
|
+
# hook for subclasses
|
78
|
+
end
|
79
|
+
|
80
|
+
def required_args
|
81
|
+
[:file_string] + subclass_required_args
|
82
|
+
end
|
83
|
+
|
84
|
+
def subclass_required_args
|
85
|
+
# hook for subclasses
|
86
|
+
[]
|
87
|
+
end
|
88
|
+
|
89
|
+
def should_replace?
|
90
|
+
@should_replace
|
91
|
+
end
|
92
|
+
|
93
|
+
def destroy_records
|
94
|
+
# hook for subclasses
|
95
|
+
end
|
96
|
+
|
97
|
+
def process_row(row, headers)
|
98
|
+
row_importer_class.new(row: row, headers: headers).import_row if row_importer_class
|
99
|
+
end
|
100
|
+
|
101
|
+
def starting_message
|
102
|
+
"Importing with #{self.class.name}...\n\n"
|
103
|
+
end
|
104
|
+
|
105
|
+
def finished_message
|
106
|
+
"Finished importing."
|
107
|
+
end
|
108
|
+
|
109
|
+
def print(message)
|
110
|
+
out.print message
|
111
|
+
end
|
112
|
+
|
113
|
+
def guard_against_errors(&block)
|
114
|
+
|
115
|
+
results = {}
|
116
|
+
|
117
|
+
begin
|
118
|
+
ActiveRecord::Base.transaction do
|
119
|
+
import_results = yield
|
120
|
+
status = import_results.any? { |result| result[:status] == Statuses::ERROR } ? Statuses::ERROR : Statuses::SUCCESS
|
121
|
+
|
122
|
+
results = {
|
123
|
+
status: status,
|
124
|
+
results: import_results
|
125
|
+
}
|
126
|
+
|
127
|
+
# need to rollback if errors
|
128
|
+
raise ActiveRecord::Rollback if status == Statuses::ERROR
|
129
|
+
end
|
130
|
+
rescue Exception => e
|
131
|
+
results = {
|
132
|
+
status: Statuses::ERROR,
|
133
|
+
error: e.message
|
134
|
+
}
|
135
|
+
end
|
136
|
+
|
137
|
+
print_results(results)
|
138
|
+
|
139
|
+
results
|
140
|
+
end
|
141
|
+
|
142
|
+
def print_results(results)
|
143
|
+
case results[:status]
|
144
|
+
when Statuses::SUCCESS
|
145
|
+
print("Imported completed successfully!".green)
|
146
|
+
when Statuses::ERROR
|
147
|
+
print("\nImported failed, all changes have been rolled back.\n\n".red)
|
148
|
+
if results[Statuses::ERROR]
|
149
|
+
print(" #{results[Statuses::ERROR]}\n\n".red)
|
150
|
+
else
|
151
|
+
results[:results].each { |result| print(" #{result}\n") }
|
152
|
+
end
|
153
|
+
end
|
154
|
+
end
|
155
|
+
|
156
|
+
def load_csv_from_file(str)
|
157
|
+
csv = CSV.parse(str, headers: true)
|
158
|
+
raise(IOError.new('There is no data to import')) if csv.length == 0
|
159
|
+
headers = csv.headers.compact
|
160
|
+
[csv, headers]
|
161
|
+
end
|
162
|
+
|
163
|
+
def parse_csv_string(csv_str, previous_results=[], &block)
|
164
|
+
|
165
|
+
csv, headers = load_csv_from_file(csv_str)
|
166
|
+
|
167
|
+
idx = 1
|
168
|
+
csv.each.map do |row|
|
169
|
+
errors = []
|
170
|
+
return_value = nil
|
171
|
+
begin
|
172
|
+
return_value = yield row, headers
|
173
|
+
rescue Exception => e
|
174
|
+
errors << e.message
|
175
|
+
end
|
176
|
+
|
177
|
+
result = previous_results.detect(lambda { {} }) { |result| result[:row] == idx+1 }
|
178
|
+
|
179
|
+
{
|
180
|
+
row: idx += 1,
|
181
|
+
status: errors.any? ? Statuses::ERROR : result.fetch(:status, Statuses::SUCCESS),
|
182
|
+
errors: result.fetch(:errors, []) + errors,
|
183
|
+
value: return_value
|
184
|
+
}
|
185
|
+
end
|
186
|
+
end
|
187
|
+
|
188
|
+
def has_errors?
|
189
|
+
results.any? { |result| result[:status] == :error }
|
190
|
+
end
|
191
|
+
end
|
192
|
+
end
|
@@ -0,0 +1,148 @@
|
|
1
|
+
require "asyncable"
|
2
|
+
|
3
|
+
module CSVImportable
|
4
|
+
module Importable
|
5
|
+
# columns: results (text), file (attachment),
|
6
|
+
# should_replace (boolean), type (string),
|
7
|
+
# status (string)
|
8
|
+
|
9
|
+
include Asyncable
|
10
|
+
extend ActiveSupport::Concern
|
11
|
+
|
12
|
+
included do
|
13
|
+
serialize :results, Hash
|
14
|
+
|
15
|
+
has_attached_file :file
|
16
|
+
validates_attachment :file, content_type: { content_type: ['text/csv']} , message: "is not in CSV format"
|
17
|
+
end
|
18
|
+
|
19
|
+
DEFAULT_BIG_FILE_THRESHOLD = 10
|
20
|
+
|
21
|
+
# === PUBLIC INTERFACE METHODS ===
|
22
|
+
def read_file
|
23
|
+
# returns CSV StringIO data
|
24
|
+
# e.g. Paperclip.io_adapters.for(file).read
|
25
|
+
fail "read_file method is required by #{self.class.name}"
|
26
|
+
end
|
27
|
+
|
28
|
+
|
29
|
+
def save_to_db
|
30
|
+
return save if respond_to?(:save)
|
31
|
+
fail "please implement the save_to_db method on #{self.class.name}"
|
32
|
+
end
|
33
|
+
# === END INTERFACE METHODS ===
|
34
|
+
|
35
|
+
def import!
|
36
|
+
# start_async provided by Asyncable module
|
37
|
+
return start_async if run_async?
|
38
|
+
process_now
|
39
|
+
end
|
40
|
+
|
41
|
+
def run_async?
|
42
|
+
big_file?
|
43
|
+
end
|
44
|
+
|
45
|
+
def not_async?
|
46
|
+
!run_async?
|
47
|
+
end
|
48
|
+
|
49
|
+
def formatted_errors
|
50
|
+
@formatted_errors ||= (
|
51
|
+
errors = []
|
52
|
+
error_key = CSVImportable::CSVImporter::Statuses::ERROR
|
53
|
+
errors << results.fetch(error_key) if results.has_key?(error_key)
|
54
|
+
errors + results.fetch(:results, [])
|
55
|
+
.select { |result| result.fetch(:status) == error_key }
|
56
|
+
.map { |error| "Line #{error.fetch(:row)}: #{error.fetch(:errors).join(', ')}" }
|
57
|
+
)
|
58
|
+
end
|
59
|
+
|
60
|
+
def display_status
|
61
|
+
case status
|
62
|
+
when Statuses::SUCCEEDED
|
63
|
+
'Import Succeeded'
|
64
|
+
when Statuses::FAILED
|
65
|
+
'Import Failed with Errors'
|
66
|
+
else
|
67
|
+
'Processing'
|
68
|
+
end
|
69
|
+
end
|
70
|
+
|
71
|
+
def number_of_records_imported
|
72
|
+
return 0 unless results && results.is_a?(Hash)
|
73
|
+
results[:results].try(:count) || 0
|
74
|
+
end
|
75
|
+
|
76
|
+
private
|
77
|
+
|
78
|
+
# === PRIVATE INTERFACE METHODS ===
|
79
|
+
def importer_class
|
80
|
+
# hook for subclasses
|
81
|
+
CSVImportable::CSVImporter
|
82
|
+
end
|
83
|
+
|
84
|
+
def row_importer_class
|
85
|
+
fail "row_importer_class class method is required by #{self.class.name}"
|
86
|
+
end
|
87
|
+
|
88
|
+
def importable_class
|
89
|
+
self.class
|
90
|
+
end
|
91
|
+
|
92
|
+
def importer_options
|
93
|
+
# hook for additional options
|
94
|
+
{}
|
95
|
+
end
|
96
|
+
# === END INTERFACE METHODS ===
|
97
|
+
|
98
|
+
def async_operation
|
99
|
+
run_importer
|
100
|
+
end
|
101
|
+
|
102
|
+
def process_now
|
103
|
+
run_importer
|
104
|
+
complete!
|
105
|
+
end
|
106
|
+
|
107
|
+
def async_complete!
|
108
|
+
# async_complete! is a hook from Asyncable module
|
109
|
+
complete!
|
110
|
+
end
|
111
|
+
|
112
|
+
def complete!
|
113
|
+
return success! if importer.succeeded?
|
114
|
+
failed!(results[:error])
|
115
|
+
end
|
116
|
+
|
117
|
+
def big_file?
|
118
|
+
importer.big_file?
|
119
|
+
end
|
120
|
+
|
121
|
+
def big_file_threshold
|
122
|
+
DEFAULT_BIG_FILE_THRESHOLD
|
123
|
+
end
|
124
|
+
|
125
|
+
def importer
|
126
|
+
return @importer if @importer
|
127
|
+
|
128
|
+
args = {
|
129
|
+
should_replace: should_replace?,
|
130
|
+
row_importer_class: row_importer_class,
|
131
|
+
big_file_threshold: big_file_threshold
|
132
|
+
}.merge(importer_options)
|
133
|
+
|
134
|
+
if new_record? && !processing? # e.g. new record that's not async
|
135
|
+
args = args.merge(file_string: read_file)
|
136
|
+
else
|
137
|
+
args = args.merge(import_id: id, importable_class: importable_class)
|
138
|
+
end
|
139
|
+
|
140
|
+
@importer ||= importer_class.new(args)
|
141
|
+
end
|
142
|
+
|
143
|
+
def run_importer
|
144
|
+
importer.import
|
145
|
+
self.results = importer.results
|
146
|
+
end
|
147
|
+
end
|
148
|
+
end
|
@@ -0,0 +1,22 @@
|
|
1
|
+
module CSVImportable
|
2
|
+
class RowImporter
|
3
|
+
include CSVImportable::CSVCoercion
|
4
|
+
attr_reader :row, :headers
|
5
|
+
|
6
|
+
def initialize(args = {})
|
7
|
+
@row = args[:row]
|
8
|
+
@headers = args[:headers]
|
9
|
+
after_init(args)
|
10
|
+
end
|
11
|
+
|
12
|
+
def import_row
|
13
|
+
# hook for subclasses
|
14
|
+
end
|
15
|
+
|
16
|
+
private
|
17
|
+
|
18
|
+
def after_init(args)
|
19
|
+
# hook for subclasses
|
20
|
+
end
|
21
|
+
end
|
22
|
+
end
|
@@ -0,0 +1,77 @@
|
|
1
|
+
module CSVImportable
|
2
|
+
class TypeParser
|
3
|
+
attr_reader :row, :key, :value, :required
|
4
|
+
|
5
|
+
def initialize(key, args = {})
|
6
|
+
@key = key
|
7
|
+
@row = args[:row]
|
8
|
+
@required = args.fetch(:required, false)
|
9
|
+
@value = args.fetch(:value, pull_value_from_row)
|
10
|
+
after_init(args)
|
11
|
+
end
|
12
|
+
|
13
|
+
def parse
|
14
|
+
if value.blank?
|
15
|
+
check_required
|
16
|
+
return nil
|
17
|
+
end
|
18
|
+
parsed_val = nil
|
19
|
+
begin
|
20
|
+
parsed_val = parse_val
|
21
|
+
rescue
|
22
|
+
raise_parsing_error
|
23
|
+
end
|
24
|
+
raise_parsing_error if parsed_val.nil?
|
25
|
+
parsed_val
|
26
|
+
end
|
27
|
+
|
28
|
+
private
|
29
|
+
|
30
|
+
def after_init(args)
|
31
|
+
# hook for subclasses
|
32
|
+
end
|
33
|
+
|
34
|
+
def pull_value_from_row
|
35
|
+
return nil unless row
|
36
|
+
# handle both caps and lowercase
|
37
|
+
row.field(key.upcase) || row.field(key.downcase)
|
38
|
+
end
|
39
|
+
|
40
|
+
def parse_val
|
41
|
+
# hook for subclasses
|
42
|
+
fail 'parse_val is a required method for subclass'
|
43
|
+
end
|
44
|
+
|
45
|
+
def raise_parsing_error
|
46
|
+
raise error_message
|
47
|
+
end
|
48
|
+
|
49
|
+
def error_message
|
50
|
+
fail 'error_message is a requird method for subclass'
|
51
|
+
end
|
52
|
+
|
53
|
+
def raise_required_error
|
54
|
+
raise required_error_message
|
55
|
+
end
|
56
|
+
|
57
|
+
def required_error_message
|
58
|
+
"#{key} is blank"
|
59
|
+
end
|
60
|
+
|
61
|
+
def required?
|
62
|
+
required
|
63
|
+
end
|
64
|
+
|
65
|
+
def check_required
|
66
|
+
return raise_required_error if required?
|
67
|
+
end
|
68
|
+
end
|
69
|
+
end
|
70
|
+
|
71
|
+
require_relative './type_parser/boolean_type_parser'
|
72
|
+
require_relative './type_parser/date_type_parser'
|
73
|
+
require_relative './type_parser/float_type_parser'
|
74
|
+
require_relative './type_parser/integer_type_parser'
|
75
|
+
require_relative './type_parser/percent_type_parser'
|
76
|
+
require_relative './type_parser/select_type_parser'
|
77
|
+
require_relative './type_parser/string_type_parser'
|
@@ -0,0 +1,13 @@
|
|
1
|
+
module CSVImportable
|
2
|
+
class TypeParser::BooleanTypeParser < TypeParser
|
3
|
+
def parse_val
|
4
|
+
val = true if ["yes", "y", true, "true"].include?(value.downcase)
|
5
|
+
val = false if ["no", "n", false, "false"].include?(value.downcase)
|
6
|
+
val
|
7
|
+
end
|
8
|
+
|
9
|
+
def error_message
|
10
|
+
"Invalid boolean for column: #{key}"
|
11
|
+
end
|
12
|
+
end
|
13
|
+
end
|
@@ -0,0 +1,23 @@
|
|
1
|
+
module CSVImportable
|
2
|
+
class TypeParser::DateTypeParser < TypeParser
|
3
|
+
def parse_val
|
4
|
+
Date.new(year, month, day)
|
5
|
+
end
|
6
|
+
|
7
|
+
def year
|
8
|
+
value[0..3].try(:to_i)
|
9
|
+
end
|
10
|
+
|
11
|
+
def month
|
12
|
+
value[4..5].try(:to_i)
|
13
|
+
end
|
14
|
+
|
15
|
+
def day
|
16
|
+
value[6..7].try(:to_i)
|
17
|
+
end
|
18
|
+
|
19
|
+
def error_message
|
20
|
+
"Invalid date for column: #{key}"
|
21
|
+
end
|
22
|
+
end
|
23
|
+
end
|
@@ -0,0 +1,22 @@
|
|
1
|
+
module CSVImportable
|
2
|
+
class TypeParser::PercentTypeParser < TypeParser
|
3
|
+
def parse_val
|
4
|
+
val = parse_percentage_sign if value.to_s.include?("%")
|
5
|
+
val = val.present? ? val : Float(value)
|
6
|
+
outside_range if val < 0 || val > 1
|
7
|
+
val
|
8
|
+
end
|
9
|
+
|
10
|
+
def error_message
|
11
|
+
"Invalid percent for column: #{key}. It should be a decimal between 0 and 1."
|
12
|
+
end
|
13
|
+
|
14
|
+
def outside_range
|
15
|
+
raise
|
16
|
+
end
|
17
|
+
|
18
|
+
def parse_percentage_sign
|
19
|
+
value.to_f / 100.0
|
20
|
+
end
|
21
|
+
end
|
22
|
+
end
|
@@ -0,0 +1,19 @@
|
|
1
|
+
module CSVImportable
|
2
|
+
class TypeParser::SelectTypeParser < TypeParser
|
3
|
+
attr_reader :options
|
4
|
+
|
5
|
+
def after_init(args = {})
|
6
|
+
@options = args.fetch(:options, [])
|
7
|
+
end
|
8
|
+
|
9
|
+
def parse_val
|
10
|
+
val = value.downcase
|
11
|
+
raise unless options.include?(val)
|
12
|
+
val
|
13
|
+
end
|
14
|
+
|
15
|
+
def error_message
|
16
|
+
"Invalid value for column: #{key}. Must be one of the following: #{options.join(', ')}"
|
17
|
+
end
|
18
|
+
end
|
19
|
+
end
|
metadata
ADDED
@@ -0,0 +1,124 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: csv_importable
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.1.0
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- Ryan Francis
|
8
|
+
autorequire:
|
9
|
+
bindir: exe
|
10
|
+
cert_chain: []
|
11
|
+
date: 2017-01-12 00:00:00.000000000 Z
|
12
|
+
dependencies:
|
13
|
+
- !ruby/object:Gem::Dependency
|
14
|
+
name: bundler
|
15
|
+
requirement: !ruby/object:Gem::Requirement
|
16
|
+
requirements:
|
17
|
+
- - "~>"
|
18
|
+
- !ruby/object:Gem::Version
|
19
|
+
version: '1.12'
|
20
|
+
type: :development
|
21
|
+
prerelease: false
|
22
|
+
version_requirements: !ruby/object:Gem::Requirement
|
23
|
+
requirements:
|
24
|
+
- - "~>"
|
25
|
+
- !ruby/object:Gem::Version
|
26
|
+
version: '1.12'
|
27
|
+
- !ruby/object:Gem::Dependency
|
28
|
+
name: rake
|
29
|
+
requirement: !ruby/object:Gem::Requirement
|
30
|
+
requirements:
|
31
|
+
- - "~>"
|
32
|
+
- !ruby/object:Gem::Version
|
33
|
+
version: '10.0'
|
34
|
+
type: :development
|
35
|
+
prerelease: false
|
36
|
+
version_requirements: !ruby/object:Gem::Requirement
|
37
|
+
requirements:
|
38
|
+
- - "~>"
|
39
|
+
- !ruby/object:Gem::Version
|
40
|
+
version: '10.0'
|
41
|
+
- !ruby/object:Gem::Dependency
|
42
|
+
name: rspec
|
43
|
+
requirement: !ruby/object:Gem::Requirement
|
44
|
+
requirements:
|
45
|
+
- - "~>"
|
46
|
+
- !ruby/object:Gem::Version
|
47
|
+
version: '3.0'
|
48
|
+
type: :development
|
49
|
+
prerelease: false
|
50
|
+
version_requirements: !ruby/object:Gem::Requirement
|
51
|
+
requirements:
|
52
|
+
- - "~>"
|
53
|
+
- !ruby/object:Gem::Version
|
54
|
+
version: '3.0'
|
55
|
+
- !ruby/object:Gem::Dependency
|
56
|
+
name: asyncable
|
57
|
+
requirement: !ruby/object:Gem::Requirement
|
58
|
+
requirements:
|
59
|
+
- - ">="
|
60
|
+
- !ruby/object:Gem::Version
|
61
|
+
version: '0'
|
62
|
+
type: :runtime
|
63
|
+
prerelease: false
|
64
|
+
version_requirements: !ruby/object:Gem::Requirement
|
65
|
+
requirements:
|
66
|
+
- - ">="
|
67
|
+
- !ruby/object:Gem::Version
|
68
|
+
version: '0'
|
69
|
+
description: Equipped with intelligent parsing rules and excellent error handling,
|
70
|
+
this gem is all you need for importing CSVs in Ruby.
|
71
|
+
email:
|
72
|
+
- ryan@launchpadlab.com
|
73
|
+
executables: []
|
74
|
+
extensions: []
|
75
|
+
extra_rdoc_files: []
|
76
|
+
files:
|
77
|
+
- ".gitignore"
|
78
|
+
- ".rspec"
|
79
|
+
- ".travis.yml"
|
80
|
+
- Gemfile
|
81
|
+
- README.md
|
82
|
+
- Rakefile
|
83
|
+
- bin/console
|
84
|
+
- bin/setup
|
85
|
+
- csv_importable.gemspec
|
86
|
+
- lib/csv_importable.rb
|
87
|
+
- lib/csv_importable/csv_coercion.rb
|
88
|
+
- lib/csv_importable/csv_importer.rb
|
89
|
+
- lib/csv_importable/importable.rb
|
90
|
+
- lib/csv_importable/row_importer.rb
|
91
|
+
- lib/csv_importable/type_parser.rb
|
92
|
+
- lib/csv_importable/type_parser/boolean_type_parser.rb
|
93
|
+
- lib/csv_importable/type_parser/date_type_parser.rb
|
94
|
+
- lib/csv_importable/type_parser/float_type_parser.rb
|
95
|
+
- lib/csv_importable/type_parser/integer_type_parser.rb
|
96
|
+
- lib/csv_importable/type_parser/percent_type_parser.rb
|
97
|
+
- lib/csv_importable/type_parser/select_type_parser.rb
|
98
|
+
- lib/csv_importable/type_parser/string_type_parser.rb
|
99
|
+
- lib/csv_importable/version.rb
|
100
|
+
homepage: https://github.com/LaunchPadLab/csv_importable
|
101
|
+
licenses: []
|
102
|
+
metadata:
|
103
|
+
allowed_push_host: https://rubygems.org
|
104
|
+
post_install_message:
|
105
|
+
rdoc_options: []
|
106
|
+
require_paths:
|
107
|
+
- lib
|
108
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
109
|
+
requirements:
|
110
|
+
- - ">="
|
111
|
+
- !ruby/object:Gem::Version
|
112
|
+
version: '0'
|
113
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
114
|
+
requirements:
|
115
|
+
- - ">="
|
116
|
+
- !ruby/object:Gem::Version
|
117
|
+
version: '0'
|
118
|
+
requirements: []
|
119
|
+
rubyforge_project:
|
120
|
+
rubygems_version: 2.5.1
|
121
|
+
signing_key:
|
122
|
+
specification_version: 4
|
123
|
+
summary: Import CSV files from users
|
124
|
+
test_files: []
|