ar_loader 0.0.6 → 0.0.8

Sign up to get free protection for your applications and to get access to all the features.
Files changed (53) hide show
  1. data/LICENSE +9 -9
  2. data/README.markdown +268 -221
  3. data/Rakefile +76 -76
  4. data/lib/VERSION +1 -1
  5. data/lib/ar_loader.rb +87 -66
  6. data/lib/ar_loader/exceptions.rb +2 -0
  7. data/lib/{engine → ar_loader}/file_definitions.rb +353 -353
  8. data/lib/{engine → ar_loader}/mapping_file_definitions.rb +87 -87
  9. data/lib/ar_loader/method_detail.rb +257 -0
  10. data/lib/ar_loader/method_mapper.rb +213 -0
  11. data/lib/helpers/jruby/jexcel_file.rb +187 -0
  12. data/lib/{engine → helpers/jruby}/word.rb +79 -70
  13. data/lib/helpers/spree_helper.rb +85 -0
  14. data/lib/loaders/csv_loader.rb +87 -0
  15. data/lib/loaders/excel_loader.rb +132 -0
  16. data/lib/loaders/loader_base.rb +205 -73
  17. data/lib/loaders/spree/image_loader.rb +45 -41
  18. data/lib/loaders/spree/product_loader.rb +140 -91
  19. data/lib/to_b.rb +24 -24
  20. data/spec/csv_loader_spec.rb +27 -0
  21. data/spec/database.yml +19 -6
  22. data/spec/db/migrate/20110803201325_create_test_bed.rb +78 -0
  23. data/spec/excel_loader_spec.rb +113 -98
  24. data/spec/fixtures/BadAssociationName.xls +0 -0
  25. data/spec/fixtures/DemoNegativeTesting.xls +0 -0
  26. data/spec/fixtures/DemoTestModelAssoc.xls +0 -0
  27. data/spec/fixtures/ProjectsMultiCategories.xls +0 -0
  28. data/spec/fixtures/SimpleProjects.xls +0 -0
  29. data/spec/fixtures/SpreeProducts.xls +0 -0
  30. data/spec/fixtures/SpreeZoneExample.csv +5 -0
  31. data/spec/fixtures/SpreeZoneExample.xls +0 -0
  32. data/spec/loader_spec.rb +116 -0
  33. data/spec/logs/test.log +5000 -0
  34. data/spec/method_mapper_spec.rb +222 -0
  35. data/spec/models.rb +55 -0
  36. data/spec/spec_helper.rb +85 -18
  37. data/spec/spree_loader_spec.rb +223 -157
  38. data/tasks/config/seed_fu_product_template.erb +15 -15
  39. data/tasks/config/tidy_config.txt +12 -12
  40. data/tasks/db_tasks.rake +64 -64
  41. data/tasks/excel_loader.rake +63 -113
  42. data/tasks/file_tasks.rake +36 -37
  43. data/tasks/loader.rake +45 -0
  44. data/tasks/spree/image_load.rake +108 -107
  45. data/tasks/spree/product_loader.rake +49 -107
  46. data/tasks/word_to_seedfu.rake +166 -166
  47. metadata +66 -61
  48. data/lib/engine/jruby/jexcel_file.rb +0 -182
  49. data/lib/engine/jruby/method_mapper_excel.rb +0 -44
  50. data/lib/engine/method_detail.rb +0 -140
  51. data/lib/engine/method_mapper.rb +0 -157
  52. data/lib/engine/method_mapper_csv.rb +0 -28
  53. data/spec/db/migrate/20110803201325_create_testbed.rb +0 -25
data/LICENSE CHANGED
@@ -1,9 +1,9 @@
1
- == ARLoader
2
-
3
- Copyright tom statter @ www.autotelik.co.uk 2011
4
-
5
- Not sure what is appropriate LICENSE
6
-
7
- Free to use in any project, commercial or otherwise as long as copyright not removed ??
8
-
9
- MIT ?
1
+ == ARLoader
2
+
3
+ Copyright tom statter @ www.autotelik.co.uk 2011
4
+
5
+ Not sure what is appropriate LICENSE
6
+
7
+ Free to use in any project, commercial or otherwise as long as copyright not removed ??
8
+
9
+ MIT ?
data/README.markdown CHANGED
@@ -1,222 +1,269 @@
1
- # AR Loader
2
-
3
- General Loader for populating database with seed data from various sources.
4
-
5
- Simplifies the specification and loading of data from files into any active record supported database.
6
-
7
- Seamlessly handles loading an active record model's attributes and it's associations,
8
- based on reflection against the supplied model - mapping column headings to attributes and associations.
9
-
10
- Fully extendable via spreadsheet headings - simply add new column to Excel/Open Office spreadsheets with
11
- attribute or association name, and loader will attempt to find correct association and populate AR object with row data.
12
-
13
- Original focus was on support for the Open Source Spree e-commerce project, so includes specific loaders and rake tasks
14
- for loading Spree Products and associated data such as Product Variants, and images.
15
-
16
- Loaders attempt to handle various human read-able forms of column names.
17
-
18
- For example, given an association on the model called, product_properties, will successfully load
19
- from columns with headings such as 'product_properties', 'Product Properties', 'product Properties' etc
20
-
21
- ## Installation
22
-
23
- Add gem instruction to your Gemfile. To use the Excel loader, JRuby is required, so to use in a mixed setup
24
- of JRuby and deployed to other Rubies, use the following guard.
25
-
26
- if(RUBY_PLATFORM =~ /java/)
27
- gem 'activerecord-jdbcmysql-adapter'
28
- else
29
- gem 'mysql'
30
- end
31
-
32
- Install the latest gem :
33
-
34
- `gem install ar_loader`
35
-
36
- To use :
37
-
38
- require 'ar_loader'
39
-
40
- To pull the tasks in, add call in your Rakefile :
41
-
42
- ArLoader::require_tasks
43
-
44
- ## Example Spreadsheet
45
-
46
- An example Spreadsheet with headers and comments, suitable for giving to Clients
47
- to populate, can be found in test/examples/DemoSpreadsheet.xls
48
-
49
- ## Features
50
-
51
- - *Direct Excel file support*
52
-
53
- Includes a wrapper around MS Excel File format, via Apache POI, which
54
- enables Products to be loaded directly from Excel files (Excel does not need to be installed) via JRuby.
55
- No need to save to CSV first.
56
-
57
- The java jars e.g - 'poi-3.6.jar' - are included.
58
-
59
- - *Semi-Smart Name Lookup*
60
-
61
- Includes helper classes that find and store details of all possible associations on an AR class.
62
- Given a user supplied name, attempts to find the requested association.
63
-
64
- Example usage, load from a file or spreadsheet where the column names are only
65
- an approximation of the actual associations, so given 'Product Properties' heading,
66
- finds real association 'product_properties' to send or call on the AR object
67
-
68
- - *Associations*
69
-
70
- Can handle 'many' type associations and enables multiple association objects to
71
- be added via single entry (column). See Details section.
72
-
73
- - *Spree Rake Tasks*
74
-
75
- Rake tasks provided for Spree loading - currently supports Product with associations,
76
- and Image loading.
77
-
78
- **Product loading from Excel specifically requires JRuby**.
79
-
80
- Example command lines:
81
-
82
- rake excel_load input=vendor\extensions\autotelik\fixtures\ExampleInfoWeb.xls
83
-
84
- rake excel_load input=C:\MyProducts.xls verbose=true'
85
-
86
- - *Seamless Image loading can be achieved by ensuring SKU or class Name features in Image filename.
87
-
88
- Lookup is performed either via the SKU being prepended to the image name, or by the image name being equal to the **name attribute** of the klass in question.
89
-
90
- Images can be attached to any class defined with a suitable association. The class to use can be configured in rake task via
91
- parameter klass=Xyz.
92
-
93
- In the Spree tasks, this defaults to Product, so attempts to attach Image to a Product via Product SKU or Name.
94
-
95
- Image loading **does not** specifically require JRuby
96
-
97
- A report is generated in the current working directory detailing any Images in the paths that could not be matched with a Product.
98
-
99
- Example cmd lines :
100
-
101
- rake image_load input=vendor\extensions\autotelik\lib\fixtures
102
- rake image_load input="C:\images\Paintings' dummy=true
103
- rake image_load input="C:\images\TaxonIcons" skip_if_no_assoc=true klass=Taxon
104
-
105
- ## Example Wrapper Tasks for Spree Site Extension
106
-
107
- These tasks show how to write your own high level wrapper task, that will seed the database from multiple spreedsheets.
108
-
109
- The images in this example have been named with the SKU present in name (separated by whitespace) e.g "PRINT_001 Stonehenge.jpg"
110
-
111
- A report is generated in the current working directory detailing any Images in the paths that could not be matched with a Product.
112
-
113
- require 'ar_loader'
114
-
115
- namespace :mysite do
116
-
117
- desc "Load Products for site"
118
- task :load, :needs => [:environment] do |t, args|
119
-
120
- [ "vendor/extensions/site/db/seed/Paintings.xls",
121
- "vendor/extensions/site/db/seed/Drawings.xls"
122
- ].each do |x|
123
- Rake::Task['autotelik:excel_load'].execute(
124
- :input => x,
125
- :verbose => true,
126
- :sku_prefix => ""
127
- )
128
- end
129
- end
130
-
131
- desc "Load Images for site based on SKU"
132
- task :load_images, :clean, :dummy, :needs => [:environment] do |t, args|
133
-
134
- if(args[:clean])
135
- Image.delete_all
136
- FileUtils.rm_rf( "public/assests/products" )
137
- end
138
-
139
- ["01_paintings_jpegs", "02_drawings_jpegs"].each do |x|
140
-
141
- # image names start with associated Product SKU,
142
- # skip rather then exit if no matching product found
143
-
144
- Rake::Task['autotelik:image_load'].execute(
145
- :input => "/my_site_load_info//#{x}",
146
- :dummy => args[:dummy],
147
- :verbose => false, :sku => true, :skip_if_no_assoc => true
148
- )
149
- end
150
- end
151
-
152
- ## Details
153
-
154
- ### Associations
155
-
156
- A single association column can contain multiple name/value sets in default form :
157
-
158
- Name1:value1, value2|Name2:value1, value2, value3|Name3:value1, value2 etc
159
-
160
- So for example a Column for an 'Option Types' association on a Product,
161
- could contain 2 options with a number of values each :
162
-
163
- 'Option Types'
164
- size:small,medium,large|colour:red,white
165
- size:small|colour:blue,red,white
166
-
167
- ### Properties
168
-
169
- The properties to associate with this product.
170
- Properties are for small snippets of text, shared across many products,
171
- and are for display purposes only.
172
-
173
- An optional display value can be supplied to supplement the displayed text.
174
-
175
- As for all associations can contain multiple name/value sets in default form :
176
-
177
- Property:display_value|Property:display_value
178
-
179
- Example - No values :
180
- manufacturer|standard
181
-
182
- Example - Display values :
183
- manufacturer:somebody else plc|standard:ISOBlah21
184
-
185
- ## TODO
186
-
187
- - Make more generic, so have smart switching to Ruby and directly support csv,
188
- when JRuby and/or Excel not available.
189
-
190
- - Smart sorting of column processing order ....
191
-
192
- Does not currently ensure mandatory columns (for valid?) processed first.
193
- Since Product needs saving before associations can be processed, user currently
194
- needs to ensure SKU, name, price columns are among first columns
195
-
196
- ## License
197
-
198
- Copyright:: (c) Autotelik Media Ltd 2011
199
-
200
- Author :: Tom Statter
201
-
202
- Date :: Feb 2011
203
-
204
- The MIT License
205
-
206
- Permission is hereby granted, free of charge, to any person obtaining a copy
207
- of this software and associated documentation files (the "Software"), to deal
208
- in the Software without restriction, including without limitation the rights
209
- to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
210
- copies of the Software, and to permit persons to whom the Software is
211
- furnished to do so, subject to the following conditions:
212
-
213
- The above copyright notice and this permission notice shall be included in
214
- all copies or substantial portions of the Software.
215
-
216
- THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
217
- IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
218
- FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
219
- AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
220
- LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
221
- OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
1
+ ## AR Loader
2
+
3
+ General active record loader for populating database with seed data from various sources,
4
+ including csv files and .xls files (Excel Spreadsheets)
5
+
6
+ Simplifies the specification and loading of data from such files into any active record supported database.
7
+
8
+ Aims to generically and seamlessly, handle loading an active record model's attributes and it's associations,
9
+ based on reflection against the supplied model.
10
+
11
+ So rather than hard coded mappings, uses the file's column headings to map data to a model's attributes and associations.
12
+
13
+ This makes loaders extendable via column/file data rather than requiring new Ruby coding.
14
+
15
+ Simply add the new column to an Excel/Open Office spreadsheet, or CSV file, and add the new
16
+ attribute or association name to the header row. Loader will attempt to find correct association and populate AR object with row data.
17
+
18
+ The Loader attempts to handle various human read-able forms of column names.
19
+
20
+ For example, given an association on the model called, product_properties, will successfully load
21
+ from columns with headings such as 'product_properties', 'Product Properties', 'ProductProperties' 'product properties' etc
22
+
23
+ For has_many associations, either multiple columns can be used or multiple values can be specified in a single column using suitable delimiters.
24
+
25
+ Complex associations/mappings, for example requiring complex lookups, can be handled by extending the loader engine.
26
+
27
+ Original focus was on support for the Open Source Spree e-commerce project, so includes specific loaders and rake tasks
28
+ for loading Spree Products, and associated data such as Product Variants, and Images.
29
+
30
+ ## Installation
31
+
32
+ Add gem 'ar_loader' to your Gemfile/bundle, or install the latest gem as usual :
33
+
34
+ `gem install ar_loader`
35
+
36
+ To use :
37
+
38
+ gem 'ar_loader'
39
+ require 'ar_loader'
40
+
41
+ ArLoader::load_tasks
42
+
43
+
44
+ To pull the tasks in, add call in your Rakefile :
45
+
46
+ ArLoader::load_tasks
47
+
48
+ N.B - To use the Excel loader, OLE and Excel are NOT required, however
49
+ JRuby is required, since it uses Java's Apache POI under the hood to process .xls files.
50
+
51
+ To use in a mixed Ruby setup, you can use a guard something like :
52
+
53
+ if(RUBY_PLATFORM =~ /java/)
54
+ gem 'activerecord-jdbcmysql-adapter'
55
+ else
56
+ gem 'mysql'
57
+ end
58
+
59
+ ## Example Spreadsheet
60
+
61
+ A number of example Spreadsheets with headers and comments, can be found in the spec/fixtures directory.
62
+
63
+
64
+ ## Features
65
+
66
+ - *Direct Excel file support*
67
+
68
+ Includes a wrapper around MS Excel File format, via Apache POI, which
69
+ enables Products to be loaded directly from Excel files (Excel does not need to be installed) via JRuby.
70
+ No need to save to CSV first.
71
+
72
+ The java jars e.g - 'poi-3.6.jar' - are included.
73
+
74
+ - *Semi-Smart Name Lookup*
75
+
76
+ Includes helper classes that find and store details of all possible associations on an AR class.
77
+ Given a user supplied name, attempts to find the requested association.
78
+
79
+ Example usage, load from a file or spreadsheet where the column names are only
80
+ an approximation of the actual associations, so given 'Product Properties' heading,
81
+ finds real association 'product_properties' to send or call on the AR object
82
+
83
+ - *Associations*
84
+
85
+ Can handle 'belongs_to, 'has_many' and 'has_one' associations, including assignment of multiple objects
86
+ via either multiple columns, or via specially delimited entry in a single (column). See Details section.
87
+
88
+
89
+ - *Rake Tasks*
90
+
91
+ High level Rake tasks are provided, only required to supply model class, and file location :
92
+
93
+ jruby -S rake ar_loader:excel model=MusicTrack input=MyTrackListing.xls
94
+
95
+
96
+ - *Spree Rake Tasks*
97
+
98
+ Specific Rake tasks are also provided for Spree loading - currently supports Product with associations,
99
+ and Image loading.
100
+
101
+ jruby -S rake ar_loader:spree:products input=C:\MyProducts.xls
102
+
103
+
104
+ **Product loading from Excel files specifically requires JRuby (But not Excel or OLE)**.
105
+
106
+
107
+ - *Seamless Spree Image loading can be achieved by ensuring SKU or class Name features in Image filename.
108
+
109
+ Lookup is performed either via the SKU being prepended to the image name, or by the image name being equal to the **name attribute** of the klass in question.
110
+
111
+ Images can be attached to any class defined with a suitable association. The class to use can be configured in rake task via
112
+ parameter klass=Xyz.
113
+
114
+ In the Spree tasks, this defaults to Product, so attempts to attach Image to a Product via Product SKU or Name.
115
+
116
+ Image loading **does not** specifically require JRuby
117
+
118
+ A report is generated in the current working directory detailing any Images in the paths that could not be matched with a Product.
119
+
120
+ rake ar_loader:spree:images input=C:\images\product_images skip_if_no_assoc=true
121
+
122
+ rake ar_loader:spree:images input=C:\images\taxon_icons skip_if_no_assoc=true klass=Taxon
123
+
124
+ ## Example Wrapper Tasks for Spree Site Extension
125
+
126
+ These tasks show how to write your own high level wrapper task, that will seed the database from multiple spreedsheets.
127
+
128
+ The images in this example have been named with the SKU present in name (separated by whitespace) e.g "PRINT_001 Stonehenge.jpg"
129
+
130
+ A report is generated in the current working directory detailing any Images in the paths that could not be matched with a Product.
131
+
132
+ require 'ar_loader'
133
+
134
+ namespace :mysite do
135
+
136
+ desc "Load Products for site"
137
+ task :load, :needs => [:environment] do |t, args|
138
+
139
+ [ "vendor/extensions/site/db/seed/Paintings.xls",
140
+ "vendor/extensions/site/db/seed/Drawings.xls"
141
+ ].each do |x|
142
+ Rake::Task['ar_loader:spree:products'].execute(
143
+ :input => x,
144
+ :verbose => true,
145
+ :sku_prefix => ""
146
+ )
147
+ end
148
+ end
149
+
150
+ desc "Load Images for site based on SKU"
151
+ task :load_images, :clean, :dummy, :needs => [:environment] do |t, args|
152
+
153
+ if(args[:clean])
154
+ Image.delete_all
155
+ FileUtils.rm_rf( "public/assests/products" )
156
+ end
157
+
158
+ ["01_paintings_jpegs", "02_drawings_jpegs"].each do |x|
159
+
160
+ # image names start with associated Product SKU,
161
+ # skip rather then exit if no matching product found
162
+
163
+ Rake::Task['autotelik:image_load'].execute(
164
+ :input => "/my_site_load_info//#{x}",
165
+ :dummy => args[:dummy],
166
+ :verbose => false, :sku => true, :skip_if_no_assoc => true
167
+ )
168
+ end
169
+ end
170
+
171
+ ## Details
172
+
173
+ ### Associations
174
+
175
+ To perform a lookup for an associated model, the primary column(s) must be supplied, along with required select values for those columns.
176
+
177
+ A single association column can contain multiple name/value sets, in string form :
178
+
179
+ column:lookup_key_1, lookup_key_2,...
180
+
181
+ So if our Project model has many Categories, we can supply a Category list, which is keyed on the column 'reference' with :
182
+
183
+ |Categories|
184
+ reference:category_001,category_002
185
+
186
+ During loading, a call to find_all_by_reference will be made, picking up the 2 categories with matching references,
187
+ and our Project model will contain those two i.e project.categories = [category_002,category_003]
188
+
189
+ ## Spree Suppprt
190
+
191
+ ### OptionTypes & Variants
192
+
193
+ When loaded with the Spree specific tasks, spree specific over rides are supported, such as direct s
194
+ support for OptionTypes with values
195
+
196
+ Any 'Option Types' columns can contain the OptionType to associate with the Product, plus a selection of
197
+ appropriate OptionValues to go with that Type.
198
+
199
+ For example, in a single column/row we could supply 2 OptionTypes (named, size & colour), with a selection values
200
+ (such as small, medium etc)
201
+
202
+ 'Option Types'
203
+ size:small,medium,large|colour:red,white
204
+
205
+ If no such OptionType exists, e.g size, then a new one is created with the supplied name.
206
+
207
+ Next the OptionValues are also parsed, again if no such OptionValue exists, e.g small, then a new one is created with the supplied name.
208
+
209
+ Lastly a Variant is created on each OptionValue, with price and availaable dates being copied from Master.
210
+ Currently a unique SKU is created by adding an index to the master's sku.
211
+
212
+ TODO - Enable a hash of attributes to be supplied in association columns to enable more control over creation of associated objects.
213
+
214
+ ### Properties
215
+
216
+ The properties to associate with this product.
217
+ Properties are for small snippets of text, shared across many products,
218
+ and are for display purposes only.
219
+
220
+ An optional display value can be supplied to supplement the displayed text.
221
+
222
+ As for all associations can contain multiple name/value sets in default form :
223
+
224
+ Property:display_value|Property:display_value
225
+
226
+ Example - No values :
227
+ manufacturer|standard
228
+
229
+ Example - Display values :
230
+ manufacturer:somebody else plc|standard:ISOBlah21
231
+
232
+ ## TODO
233
+
234
+ - Directly support csv,
235
+ when JRuby and/or Excel not available.
236
+
237
+ - Smart sorting of column processing order ....
238
+
239
+ Does not currently ensure mandatory columns (for valid?) processed first.
240
+ Since Product needs saving before associations can be processed, user currently
241
+ needs to ensure SKU, name, price columns are among first columns
242
+
243
+ ## License
244
+
245
+ Copyright:: (c) Autotelik Media Ltd 2011
246
+
247
+ Author :: Tom Statter
248
+
249
+ Date :: Feb 2011
250
+
251
+ The MIT License
252
+
253
+ Permission is hereby granted, free of charge, to any person obtaining a copy
254
+ of this software and associated documentation files (the "Software"), to deal
255
+ in the Software without restriction, including without limitation the rights
256
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
257
+ copies of the Software, and to permit persons to whom the Software is
258
+ furnished to do so, subject to the following conditions:
259
+
260
+ The above copyright notice and this permission notice shall be included in
261
+ all copies or substantial portions of the Software.
262
+
263
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
264
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
265
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
266
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
267
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
268
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
222
269
  THE SOFTWARE.