datashift 0.15.0 → 0.16.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (58) hide show
  1. checksums.yaml +7 -0
  2. data/README.markdown +91 -55
  3. data/VERSION +1 -1
  4. data/datashift.gemspec +8 -23
  5. data/lib/applications/jexcel_file.rb +1 -2
  6. data/lib/datashift.rb +34 -15
  7. data/lib/datashift/column_packer.rb +98 -34
  8. data/lib/datashift/data_transforms.rb +83 -0
  9. data/lib/datashift/delimiters.rb +58 -10
  10. data/lib/datashift/excel_base.rb +123 -0
  11. data/lib/datashift/exceptions.rb +45 -7
  12. data/lib/datashift/load_object.rb +25 -0
  13. data/lib/datashift/mapping_service.rb +91 -0
  14. data/lib/datashift/method_detail.rb +40 -62
  15. data/lib/datashift/method_details_manager.rb +18 -2
  16. data/lib/datashift/method_dictionary.rb +27 -10
  17. data/lib/datashift/method_mapper.rb +49 -41
  18. data/lib/datashift/model_mapper.rb +42 -22
  19. data/lib/datashift/populator.rb +258 -143
  20. data/lib/datashift/thor_base.rb +38 -0
  21. data/lib/exporters/csv_exporter.rb +57 -145
  22. data/lib/exporters/excel_exporter.rb +73 -60
  23. data/lib/generators/csv_generator.rb +65 -5
  24. data/lib/generators/generator_base.rb +69 -3
  25. data/lib/generators/mapping_generator.rb +112 -0
  26. data/lib/helpers/core_ext/csv_file.rb +33 -0
  27. data/lib/loaders/csv_loader.rb +41 -39
  28. data/lib/loaders/excel_loader.rb +130 -116
  29. data/lib/loaders/loader_base.rb +190 -146
  30. data/lib/loaders/paperclip/attachment_loader.rb +4 -4
  31. data/lib/loaders/paperclip/datashift_paperclip.rb +5 -3
  32. data/lib/loaders/paperclip/image_loading.rb +9 -7
  33. data/lib/loaders/reporter.rb +17 -8
  34. data/lib/thor/export.thor +12 -13
  35. data/lib/thor/generate.thor +1 -9
  36. data/lib/thor/import.thor +13 -24
  37. data/lib/thor/mapping.thor +65 -0
  38. data/spec/Gemfile +13 -11
  39. data/spec/Gemfile.lock +98 -93
  40. data/spec/csv_exporter_spec.rb +104 -99
  41. data/spec/csv_generator_spec.rb +159 -0
  42. data/spec/csv_loader_spec.rb +197 -16
  43. data/spec/datashift_spec.rb +9 -0
  44. data/spec/excel_exporter_spec.rb +149 -58
  45. data/spec/excel_generator_spec.rb +35 -44
  46. data/spec/excel_loader_spec.rb +196 -178
  47. data/spec/excel_spec.rb +8 -5
  48. data/spec/loader_base_spec.rb +47 -7
  49. data/spec/mapping_spec.rb +117 -0
  50. data/spec/method_dictionary_spec.rb +24 -11
  51. data/spec/method_mapper_spec.rb +5 -7
  52. data/spec/model_mapper_spec.rb +41 -0
  53. data/spec/paperclip_loader_spec.rb +3 -6
  54. data/spec/populator_spec.rb +48 -14
  55. data/spec/spec_helper.rb +85 -73
  56. data/spec/thor_spec.rb +40 -5
  57. metadata +93 -86
  58. data/lib/applications/excel_base.rb +0 -63
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: 4e73de99a2466b06c8dfb6bab01cb7c2ab00fb54
4
+ data.tar.gz: 73c47634a567004886c55e75b1f03a63a75ff155
5
+ SHA512:
6
+ metadata.gz: ec2035080b4d6cfc1560530639084e35271f2716a6772f0b8ada9d2d62752f8c3e5756babc4e5bc9afa83636b9adef4c047bc5db99ad26e31e8e75e51ff3b89d
7
+ data.tar.gz: 426c72aadaab70fa3e1a09c8b131a0b862e75d94ad138316474cf75cf9af0a27aaf7d72037f85404d94dbdab7c668b8a47f5939da56914d0c6591421fd79fd4a
@@ -1,59 +1,20 @@
1
1
  ## DataShift
2
2
 
3
+ - [Installation](#Installation)
3
4
  - [Features](#features)
4
- - [Installation](#installation)
5
- - [Active Record - Import/Export](#Active Record - Import/Export)
5
+ - [Testing](#testing)
6
6
  - [License](#license)
7
7
 
8
- Provides tools to shift data between Excel/CSV files and Rails projects and Ruby applications
9
-
10
- Import and export models fully with all associations.
8
+ Shift data between Excel/CSV files and Rails or Ruby applications
11
9
 
12
10
  Comprehensive Wiki here : **https://github.com/autotelik/datashift/wiki**
13
11
 
14
- Specific command line tools and full Product loading for Spree E-Commerce
15
- now separate gem at [datashift_spree](https://github.com/autotelik/datashift_spree "Datashift Spree")
16
-
17
-
18
- ### Features
19
-
20
- Import and Export ActiveRecord models direct to CSV or Excel/OpenOffice (.xls) (JRuby, 1.8.7, REE, 1.9.3)
21
-
22
- You can select which associations to include and for import, set configurable defaults or over rides.
23
-
24
- Create, parse and use Excel/OpenOffice (.xls) documents dynamically from Ruby (JRuby, 1.8.7, REE, 1.9.3)
25
-
26
- Generate a sample template with headers only.
27
-
28
- Export template and populate with model data
29
-
30
- Bulk import tools for Paperclip attachments.
31
-
32
- Easily extendable Loader functionality to deal with non trivial import cases, such
33
- as complex association lookups.
34
-
35
- High level rake and thor command line tasks for import/export provided.
36
-
37
- Specific loaders and command line tasks provided out the box for **Spree E-Commerce**,
38
- enabling import/export of Product data including creating Variants with different
39
- count on hands and all associations including Properties/Taxons/OptionTypes and Images.
40
-
41
- Loaders can be configured via YAML with over ride values, default values and mandatory column settings.
42
-
43
- Many example Spreadsheets/CSV files in spec/fixtures, fully documented with comments for each column.
44
-
45
- ## Installation
12
+ ### <a name="Installation">Installation</a>
46
13
 
47
14
  Add gem 'datashift' to your Gemfile/bundle or use ```gem install```
48
15
 
49
- ```ruby
50
- gem 'datashift'
51
- ```
52
-
53
- For Spree support also add :
54
-
55
- ```ruby
56
- gem 'datashift_spree'
16
+ ```ruby
17
+ gem 'datashift'
57
18
  ```
58
19
 
59
20
  To use :
@@ -64,7 +25,7 @@ To use the Thor command line applications, pull in the tasks.
64
25
 
65
26
  Generally the easiest way is to, create a high level .thor file in your Rails root directory
66
27
 
67
- e.g mysite.thor
28
+ e.g mysite.thor
68
29
 
69
30
  Edit the file and add the following to pull in the thor commands :
70
31
 
@@ -89,6 +50,33 @@ To get usage information use thor help <command>, for example
89
50
 
90
51
  To use Excel OLE and MS Excel are NOT required.
91
52
 
53
+ Specific tools for Spree E-Commerce now separate gem [datashift_spree](https://github.com/autotelik/datashift_spree "Datashift Spree")
54
+
55
+
56
+ #### <a name="Features">Features</a>
57
+
58
+ Import and Export ActiveRecord models direct to CSV or Excel/OpenOffice (.xls)
59
+
60
+ You can select which associations to include and for import, set configurable defaults or over rides.
61
+
62
+ Create, parse and use Excel/OpenOffice (.xls) documents dynamically from Ruby
63
+
64
+ Generate a sample template with headers only.
65
+
66
+ Export template and populate with model data
67
+
68
+ Bulk import tools for Paperclip attachments.
69
+
70
+ Easily extendable Loader functionality to deal with non trivial import cases, such
71
+ as complex association lookups.
72
+
73
+ High level rake and thor command line tasks for import/export provided.
74
+
75
+
76
+ Loaders can be configured via YAML with over ride values, default values and mandatory column settings.
77
+
78
+ Many example Spreadsheets/CSV files in spec/fixtures, fully documented with comments for each column.
79
+
92
80
  Features a common Excel interface over both our own wrapper around Apache POI (JRuby) and spreadsheet gem (all main Rubies)
93
81
 
94
82
  This means you can switch seamlessly between the two libraries, and if required drop down to make use of advanced
@@ -102,7 +90,7 @@ Guards are provided, and used internally, for mixed Ruby setups. Can be used lik
102
90
  ..do something with speadsheet
103
91
  end
104
92
 
105
- ## Active Record - Import/Export
93
+ #### Active Record - Import/Export
106
94
 
107
95
  Provides high level tasks for importing data via ActiveRecord models into a DB,
108
96
  from various sources, currently csv or .xls files (Excel/Open Office)
@@ -145,7 +133,7 @@ complicated lookup requirements. Spree is the prime Open Source e-commerce proje
145
133
  and the specific loaders and tasks support loading Spree Products, and associated data such as Variants,
146
134
  OptionTypes, Properties and Images.
147
135
 
148
- ## Template Generation and Export
136
+ #### Template Generation and Export
149
137
 
150
138
  Template generation tasks can be used to export a model's definition as column headings to CSV or .xls.
151
139
  These can be provided to developers or business users, as a template for data collection and then loading.
@@ -155,7 +143,7 @@ Export tasks can be used to export of a model's definition and any existing data
155
143
  This data can be exported directly to CSV or Excel/OpenOffice spreadsheets.
156
144
 
157
145
 
158
- ## Example Spreadsheets
146
+ #### Example Spreadsheets
159
147
 
160
148
  A number of example Spreadsheets with headers and comments, can be found in the spec/fixtures directory.
161
149
 
@@ -165,7 +153,7 @@ This data can be exported directly to CSV or Excel/OpenOffice spreadsheets.
165
153
  Column headings contain comments with full descriptions and instructions on syntax.
166
154
 
167
155
 
168
- ## Excel
156
+ #### Excel
169
157
 
170
158
 
171
159
  MS Excel itself does not need to be installed.
@@ -181,7 +169,7 @@ This data can be exported directly to CSV or Excel/OpenOffice spreadsheets.
181
169
  without converting first to CSV or YAML.
182
170
 
183
171
 
184
- ### Associations
172
+ #### Associations
185
173
 
186
174
  To perform a lookup for an associated model, the primary column(s) must be supplied, along with required select values for those columns.
187
175
 
@@ -199,7 +187,7 @@ During loading, a call to find_all_by_reference will be made, picking up the 2 c
199
187
  and our Project model will contain those two i.e project.categories = [category_002,category_003]
200
188
 
201
189
 
202
- ## TODO
190
+ ### TODO
203
191
 
204
192
  - Smart sorting of column processing order ....
205
193
 
@@ -208,14 +196,62 @@ During loading, a call to find_all_by_reference will be made, picking up the 2 c
208
196
  - Look at implementing import/export API using something like https://github.com/ianwhite/orm_adapter
209
197
  rather than active record, so we can support additional ORMs
210
198
 
211
-
199
+
200
+ ### <a name="Testing">Testing</a>
201
+ Specs have own Gemfile, so you can specify versions of active record that you want specs to run against :
202
+
203
+ Edit
204
+ ```ruby spec/Gemfile. ```
205
+
206
+ Then run :
207
+
208
+ ```ruby
209
+ cd spec
210
+ bundle install
211
+ ```
212
+
213
+ #### Changing Versions
214
+
215
+ A sandbox will be generated in spec/sandbox if no such directory exists.
216
+
217
+ **N.B Manual Step**
218
+ When changing versions you probably need to **delete this whole directory** spec/sandbox. Next time you run spree specs it will be auto generated using latest Rails versions
219
+
220
+ The database are created in sqlite3 and are stored in spec/fixtures. When switching versions, of say Spree,
221
+ you will probably want to and to clear out old versions and retrigger the migrations
222
+
223
+ rm spec/fixtures/*.sqlite
224
+
225
+ You will probably also want to remove lock file :
226
+
227
+ rm spec/Gemfile.lock
228
+
229
+ First time the sandbox is regenerated, alot of tests may fail,perhaps not everything loads correctly during regeneration process.
230
+
231
+ Invariably the next run, the specs pass, so a fix is low priority.
232
+
233
+ #### Run the Tests
234
+
235
+ ** N.B You should run the specs from within the specs directory. **
236
+ ```ruby
237
+ bundle exec rspec -c .
238
+ ```
239
+
240
+ A datashift **log **will be written within **spec/logs**, which hooks into the standard active record logger
241
+
242
+ /log/datashift.log
243
+ spec/logs/datashift_spec.log
244
+
245
+
246
+
247
+
212
248
  ## License
213
249
 
214
- Copyright:: (c) Autotelik Media Ltd 2011
250
+ Copyright:: (c) Autotelik Media Ltd 2015
215
251
 
216
252
  Author :: Tom Statter
217
253
 
218
- Date :: Dec 2011
254
+ Date :: Dec 2015
219
255
 
220
256
  The MIT License
221
257
 
data/VERSION CHANGED
@@ -1 +1 @@
1
- 0.15.0
1
+ 0.16.0
@@ -1,21 +1,16 @@
1
1
  require 'rake'
2
2
 
3
- lib = File.expand_path('../lib/', __FILE__)
4
-
5
- $:.unshift lib unless $:.include?(lib)
6
-
7
- require 'datashift'
3
+ #TODO version = File.read("VERSION").strip
8
4
 
9
5
  Gem::Specification.new do |s|
10
6
  s.name = "datashift"
11
- s.version = DataShift::gem_version
12
- s.date = Date.today.to_s
7
+ s.version = "0.16.0"
13
8
 
14
9
  s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
15
10
 
16
11
  s.authors = ["Thomas Statter"]
17
12
 
18
- s.description = "Comprehensive tools to import/export between Excel/CSV and ActiveRecord databases, Rails apps, and any Ruby project."
13
+ s.description = "Comprehensive import/export tools between Excel/CSV & ActiveRecord Databases, Rails apps, and any Ruby project."
19
14
  s.email = "rubygems@autotelik.co.uk"
20
15
  s.extra_rdoc_files = [
21
16
  "LICENSE.txt",
@@ -42,22 +37,12 @@ Gem::Specification.new do |s|
42
37
  s.homepage = "http://github.com/autotelik/datashift"
43
38
  s.licenses = ["MIT"]
44
39
  s.require_paths = ["lib"]
45
- s.rubygems_version = "1.8.24"
40
+
46
41
  s.summary = "Shift data betwen Excel/CSV and any Ruby app"
47
42
 
48
- if s.respond_to? :specification_version then
49
- s.specification_version = 3
50
-
51
- if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
52
- s.add_runtime_dependency(%q<spreadsheet>, [">= 0"])
53
- s.add_runtime_dependency(%q<rubyzip>, [">= 0"])
54
- else
55
- s.add_dependency(%q<spreadsheet>, [">= 0"])
56
- s.add_dependency(%q<rubyzip>, [">= 0"])
57
- end
58
- else
59
- s.add_dependency(%q<spreadsheet>, [">= 0"])
60
- s.add_dependency(%q<rubyzip>, [">= 0"])
61
- end
43
+ s.add_dependency 'spreadsheet'
44
+ s.add_dependency 'rubyzip'
45
+
46
+
62
47
  end
63
48
 
@@ -97,8 +97,7 @@ if(DataShift::Guards::jruby?)
97
97
  else
98
98
 
99
99
  name = sanitize_sheet_name( sheet_name )
100
-
101
- puts "WTF #{name}"
100
+
102
101
  if (@workbook.getSheetIndex(name) < 0) #Check sheet doesn't already exist
103
102
  return create_sheet_and_set_styles( name )
104
103
  else
@@ -25,7 +25,7 @@
25
25
 
26
26
 
27
27
  # Details:: Active Record Import/Export for .xls or CSV
28
- #
28
+ #
29
29
  # To pull DataShift commands into your main application :
30
30
  #
31
31
  # require 'datashift'
@@ -58,9 +58,9 @@ module DataShift
58
58
  def self.library_path
59
59
  File.expand_path("#{File.dirname(__FILE__)}/../lib")
60
60
  end
61
-
61
+
62
62
  def self.require_libraries
63
-
63
+
64
64
  loader_libs = %w{ lib }
65
65
 
66
66
  # Base search paths - these will be searched recursively
@@ -78,8 +78,8 @@ module DataShift
78
78
  end
79
79
  end
80
80
  end
81
-
82
- require_libs = %w{ datashift loaders helpers }
81
+
82
+ require_libs = %w{ datashift loaders generators helpers }
83
83
 
84
84
  require_libs.each do |base|
85
85
  Dir[File.join(library_path, base, '*.rb')].each do |rb|
@@ -90,6 +90,15 @@ module DataShift
90
90
  end
91
91
  end
92
92
 
93
+ if(DataShift::Guards.jruby?)
94
+ require 'jexcel_file'
95
+ JExcelFile
96
+ else
97
+ require 'spreadsheet'
98
+ require 'spreadsheet_extensions'
99
+ Spreadsheet
100
+ end
101
+
93
102
  end
94
103
 
95
104
  # Load all the datashift rake tasks and make them available throughout app
@@ -100,36 +109,46 @@ module DataShift
100
109
  Dir["#{base}/*.rake"].sort.each { |ext| load ext }
101
110
  end
102
111
 
103
-
112
+
104
113
  # Load all the datashift Thor commands and make them available throughout app
105
114
 
106
115
  def self.load_commands()
107
116
  base = File.join(library_path, 'thor', '**')
108
-
117
+
109
118
  Dir["#{base}/*.thor"].each do |f|
110
119
  next unless File.file?(f)
111
120
  Thor::Util.load_thorfile(f)
112
121
  end
113
122
  end
114
-
123
+
115
124
  end
116
125
 
126
+
127
+ require_relative 'datashift/delimiters'
128
+ require_relative 'datashift/column_packer'
129
+ require_relative 'datashift/logging'
130
+ require_relative 'datashift/exceptions'
131
+ require_relative 'datashift/guards'
132
+
133
+ require_relative 'helpers/core_ext/to_b'
134
+ require_relative 'helpers/core_ext/csv_file'
135
+
136
+ require_relative 'datashift/method_detail'
137
+ require_relative 'datashift/method_dictionary'
138
+ require_relative 'datashift/method_mapper'
139
+ require_relative 'datashift/model_mapper'
140
+
117
141
  DataShift::require_libraries
118
142
 
119
- require 'datashift/guards'
120
- require 'datashift/logging'
121
- require 'datashift/method_detail'
122
- require 'datashift/method_dictionary'
123
- require 'datashift/method_mapper'
124
143
 
125
144
  module DataShift
126
145
  if(Guards::jruby?)
127
146
  require 'java'
128
-
147
+
129
148
  class Object
130
149
  def add_to_classpath(path)
131
150
  $CLASSPATH << File.join( DataShift.root_path, 'lib', path.gsub("\\", "/") )
132
151
  end
133
152
  end
134
153
  end
135
- end
154
+ end
@@ -5,60 +5,124 @@
5
5
  #
6
6
  # Details:: Helper for creating consistent import/export format
7
7
  # of model's attributes/associations
8
- #
9
8
  #
10
- require 'exporter_base'
11
- require 'csv'
12
-
13
9
  module DataShift
14
10
 
15
11
  module ColumnPacker
16
12
 
17
-
18
- def text_delim
19
- @text_delim ||= "\'"
20
- end
13
+ include Delimiters
21
14
 
22
- def text_delim=(x)
23
- @text_delim = x
24
- end
25
-
26
15
  # Return opposite of text delim - "hello, 'barry'" => '"hello, "barry""'
27
16
  def escape_text_delim
28
17
  return '"' if text_delim == "\'"
29
18
  "\'"
30
19
  end
31
-
32
-
20
+
21
+
22
+ # Ensure a value is written to CSV correctly
23
+ # TODO - better ways ?? - see transcoding and String#encode
24
+
25
+ def escape_for_csv(value)
26
+ text = value.to_s.gsub(text_delim, escape_text_delim()).gsub("\n", "\\n")
27
+
28
+ text = "#{text_delim}#{text}#{text_delim}" if(text.include?(Delimiters::csv_delim))
29
+ text
30
+ end
31
+
32
+
33
+ def to_headers( records, associations = nil, options = {} )
34
+ return if( !records.first.is_a?(ActiveRecord::Base) || records.empty?)
35
+
36
+ only = *options[:only] ? [*options[:only]] : nil
37
+
38
+ headers =[]
39
+
40
+ if associations
41
+ details_mgr = DataShift::MethodDictionary.method_details_mgrs[records.first.class]
42
+
43
+ [*associations].each do |a|
44
+
45
+ details_mgr.get_list(a).each do |md|
46
+
47
+ next if(only && !only.include?( md.name.to_sym ) )
48
+
49
+ headers << "#{md.operator}"
50
+
51
+ end
52
+ end if(details_mgr)
53
+
54
+ else
55
+
56
+ headers = records.first.class.columns.collect( &:name )
57
+ end
58
+
59
+ headers
60
+ end
61
+
62
+
33
63
  # Convert an AR instance to a single column
34
-
35
- def record_to_column(record, attribute_delim = Delimiters::csv_delim)
36
-
37
- csv_data = []
38
- record.serializable_hash.each do |name, value|
39
- value = 'nil' if value.nil?
40
- text = value.to_s.gsub(@text_delim, escape_text_delim())
41
- csv_data << "#{name.to_sym} => #{text}"
64
+ # e.g User : ":name = > 'tom', :role => 'developer'"
65
+ #
66
+ # OPTIONS
67
+ # with_only Specify (as symbols) columns for association types to export
68
+ # json: Export association data in single column in JSON format
69
+ #
70
+ def record_to_column(record, options = {})
71
+
72
+ return "" if(record.nil? || (record.respond_to?(:each) && record.empty?) )
73
+
74
+ with_only = *options[:with_only] ? [*options[:with_only]] : nil
75
+
76
+ return record.to_json if(options[:json] && !with_only) # packs associations into single column
77
+
78
+ if( record.respond_to?(:each) )
79
+
80
+ return "" if(record.empty?)
81
+
82
+ data = []
83
+
84
+ record.each { |r| data << record_to_column(r, options); }
85
+
86
+ if(options[:json])
87
+ return data.to_json
88
+ else
89
+ return "#{data.join(Delimiters::multi_assoc_delim)}"
90
+ end
91
+
92
+ else
93
+
94
+ data = options[:json] ? {} : []
95
+
96
+ record.serializable_hash.each do |name, value|
97
+ next if(with_only && !with_only.include?( name.to_sym ) )
98
+
99
+ if(options[:json])
100
+ data[name] = value
101
+ else
102
+ data << "#{name.to_sym} #{Delimiters::key_value_sep} #{value.to_s.gsub(text_delim, escape_text_delim)}"
103
+ end
104
+ end
105
+
106
+ if(options[:json])#
107
+ return data.to_json
108
+ else
109
+ "#{Delimiters::attribute_list_start}#{data.join(Delimiters::multi_value_delim)}#{Delimiters::attribute_list_end}"
110
+ end
111
+
42
112
  end
43
- "#{csv_data.join(attribute_delim)}"
113
+
44
114
  end
45
-
46
-
115
+
116
+
47
117
  # Convert an AR instance to a set of CSV columns
48
118
  def record_to_csv(record, options = {})
49
119
  csv_data = record.serializable_hash.values.collect { |value| escape_for_csv(value) }
50
120
 
51
121
  [*options[:methods]].each { |x| csv_data << escape_for_csv(record.send(x)) if(record.respond_to?(x)) } if(options[:methods])
52
-
122
+
53
123
  csv_data.join( Delimiters::csv_delim )
54
124
  end
55
-
56
- def escape_for_csv(value)
57
- text = value.to_s.gsub(@text_delim, escape_text_delim())
58
-
59
- text = "#{@text_delim}#{text}#{@text_delim}" if(text.include?(Delimiters::csv_delim))
60
- text
61
- end
62
-
125
+
126
+
63
127
  end
64
128
  end