ar_loader 0.0.6 → 0.0.8
Sign up to get free protection for your applications and to get access to all the features.
- data/LICENSE +9 -9
- data/README.markdown +268 -221
- data/Rakefile +76 -76
- data/lib/VERSION +1 -1
- data/lib/ar_loader.rb +87 -66
- data/lib/ar_loader/exceptions.rb +2 -0
- data/lib/{engine → ar_loader}/file_definitions.rb +353 -353
- data/lib/{engine → ar_loader}/mapping_file_definitions.rb +87 -87
- data/lib/ar_loader/method_detail.rb +257 -0
- data/lib/ar_loader/method_mapper.rb +213 -0
- data/lib/helpers/jruby/jexcel_file.rb +187 -0
- data/lib/{engine → helpers/jruby}/word.rb +79 -70
- data/lib/helpers/spree_helper.rb +85 -0
- data/lib/loaders/csv_loader.rb +87 -0
- data/lib/loaders/excel_loader.rb +132 -0
- data/lib/loaders/loader_base.rb +205 -73
- data/lib/loaders/spree/image_loader.rb +45 -41
- data/lib/loaders/spree/product_loader.rb +140 -91
- data/lib/to_b.rb +24 -24
- data/spec/csv_loader_spec.rb +27 -0
- data/spec/database.yml +19 -6
- data/spec/db/migrate/20110803201325_create_test_bed.rb +78 -0
- data/spec/excel_loader_spec.rb +113 -98
- data/spec/fixtures/BadAssociationName.xls +0 -0
- data/spec/fixtures/DemoNegativeTesting.xls +0 -0
- data/spec/fixtures/DemoTestModelAssoc.xls +0 -0
- data/spec/fixtures/ProjectsMultiCategories.xls +0 -0
- data/spec/fixtures/SimpleProjects.xls +0 -0
- data/spec/fixtures/SpreeProducts.xls +0 -0
- data/spec/fixtures/SpreeZoneExample.csv +5 -0
- data/spec/fixtures/SpreeZoneExample.xls +0 -0
- data/spec/loader_spec.rb +116 -0
- data/spec/logs/test.log +5000 -0
- data/spec/method_mapper_spec.rb +222 -0
- data/spec/models.rb +55 -0
- data/spec/spec_helper.rb +85 -18
- data/spec/spree_loader_spec.rb +223 -157
- data/tasks/config/seed_fu_product_template.erb +15 -15
- data/tasks/config/tidy_config.txt +12 -12
- data/tasks/db_tasks.rake +64 -64
- data/tasks/excel_loader.rake +63 -113
- data/tasks/file_tasks.rake +36 -37
- data/tasks/loader.rake +45 -0
- data/tasks/spree/image_load.rake +108 -107
- data/tasks/spree/product_loader.rake +49 -107
- data/tasks/word_to_seedfu.rake +166 -166
- metadata +66 -61
- data/lib/engine/jruby/jexcel_file.rb +0 -182
- data/lib/engine/jruby/method_mapper_excel.rb +0 -44
- data/lib/engine/method_detail.rb +0 -140
- data/lib/engine/method_mapper.rb +0 -157
- data/lib/engine/method_mapper_csv.rb +0 -28
- data/spec/db/migrate/20110803201325_create_testbed.rb +0 -25
data/LICENSE
CHANGED
@@ -1,9 +1,9 @@
|
|
1
|
-
== ARLoader
|
2
|
-
|
3
|
-
Copyright tom statter @ www.autotelik.co.uk 2011
|
4
|
-
|
5
|
-
Not sure what is appropriate LICENSE
|
6
|
-
|
7
|
-
Free to use in any project, commercial or otherwise as long as copyright not removed ??
|
8
|
-
|
9
|
-
MIT ?
|
1
|
+
== ARLoader
|
2
|
+
|
3
|
+
Copyright tom statter @ www.autotelik.co.uk 2011
|
4
|
+
|
5
|
+
Not sure what is appropriate LICENSE
|
6
|
+
|
7
|
+
Free to use in any project, commercial or otherwise as long as copyright not removed ??
|
8
|
+
|
9
|
+
MIT ?
|
data/README.markdown
CHANGED
@@ -1,222 +1,269 @@
|
|
1
|
-
|
2
|
-
|
3
|
-
General
|
4
|
-
|
5
|
-
|
6
|
-
|
7
|
-
|
8
|
-
|
9
|
-
|
10
|
-
|
11
|
-
|
12
|
-
|
13
|
-
|
14
|
-
|
15
|
-
|
16
|
-
|
17
|
-
|
18
|
-
|
19
|
-
|
20
|
-
|
21
|
-
|
22
|
-
|
23
|
-
|
24
|
-
|
25
|
-
|
26
|
-
|
27
|
-
|
28
|
-
|
29
|
-
|
30
|
-
|
31
|
-
|
32
|
-
|
33
|
-
|
34
|
-
`gem install ar_loader`
|
35
|
-
|
36
|
-
To use :
|
37
|
-
|
38
|
-
|
39
|
-
|
40
|
-
|
41
|
-
|
42
|
-
|
43
|
-
|
44
|
-
|
45
|
-
|
46
|
-
|
47
|
-
|
48
|
-
|
49
|
-
|
50
|
-
|
51
|
-
|
52
|
-
|
53
|
-
|
54
|
-
|
55
|
-
|
56
|
-
|
57
|
-
|
58
|
-
|
59
|
-
|
60
|
-
|
61
|
-
|
62
|
-
|
63
|
-
|
64
|
-
|
65
|
-
|
66
|
-
|
67
|
-
|
68
|
-
|
69
|
-
|
70
|
-
|
71
|
-
|
72
|
-
|
73
|
-
|
74
|
-
|
75
|
-
|
76
|
-
and
|
77
|
-
|
78
|
-
|
79
|
-
|
80
|
-
|
81
|
-
|
82
|
-
|
83
|
-
|
84
|
-
|
85
|
-
|
86
|
-
|
87
|
-
|
88
|
-
|
89
|
-
|
90
|
-
|
91
|
-
|
92
|
-
|
93
|
-
|
94
|
-
|
95
|
-
|
96
|
-
|
97
|
-
|
98
|
-
|
99
|
-
|
100
|
-
|
101
|
-
|
102
|
-
|
103
|
-
|
104
|
-
|
105
|
-
|
106
|
-
|
107
|
-
|
108
|
-
|
109
|
-
|
110
|
-
|
111
|
-
|
112
|
-
|
113
|
-
|
114
|
-
|
115
|
-
|
116
|
-
|
117
|
-
|
118
|
-
|
119
|
-
|
120
|
-
|
121
|
-
|
122
|
-
|
123
|
-
|
124
|
-
|
125
|
-
|
126
|
-
|
127
|
-
|
128
|
-
|
129
|
-
|
130
|
-
|
131
|
-
|
132
|
-
|
133
|
-
|
134
|
-
|
135
|
-
|
136
|
-
|
137
|
-
|
138
|
-
|
139
|
-
["
|
140
|
-
|
141
|
-
|
142
|
-
|
143
|
-
|
144
|
-
|
145
|
-
:
|
146
|
-
|
147
|
-
|
148
|
-
|
149
|
-
|
150
|
-
|
151
|
-
|
152
|
-
|
153
|
-
|
154
|
-
|
155
|
-
|
156
|
-
|
157
|
-
|
158
|
-
|
159
|
-
|
160
|
-
|
161
|
-
|
162
|
-
|
163
|
-
'
|
164
|
-
|
165
|
-
|
166
|
-
|
167
|
-
|
168
|
-
|
169
|
-
|
170
|
-
|
171
|
-
|
172
|
-
|
173
|
-
|
174
|
-
|
175
|
-
|
176
|
-
|
177
|
-
|
178
|
-
|
179
|
-
|
180
|
-
|
181
|
-
|
182
|
-
|
183
|
-
|
184
|
-
|
185
|
-
|
186
|
-
|
187
|
-
|
188
|
-
|
189
|
-
|
190
|
-
|
191
|
-
|
192
|
-
|
193
|
-
|
194
|
-
|
195
|
-
|
196
|
-
|
197
|
-
|
198
|
-
|
199
|
-
|
200
|
-
|
201
|
-
|
202
|
-
|
203
|
-
|
204
|
-
|
205
|
-
|
206
|
-
|
207
|
-
|
208
|
-
|
209
|
-
|
210
|
-
|
211
|
-
|
212
|
-
|
213
|
-
|
214
|
-
|
215
|
-
|
216
|
-
|
217
|
-
|
218
|
-
|
219
|
-
|
220
|
-
|
221
|
-
|
1
|
+
## AR Loader
|
2
|
+
|
3
|
+
General active record loader for populating database with seed data from various sources,
|
4
|
+
including csv files and .xls files (Excel Spreadsheets)
|
5
|
+
|
6
|
+
Simplifies the specification and loading of data from such files into any active record supported database.
|
7
|
+
|
8
|
+
Aims to generically and seamlessly, handle loading an active record model's attributes and it's associations,
|
9
|
+
based on reflection against the supplied model.
|
10
|
+
|
11
|
+
So rather than hard coded mappings, uses the file's column headings to map data to a model's attributes and associations.
|
12
|
+
|
13
|
+
This makes loaders extendable via column/file data rather than requiring new Ruby coding.
|
14
|
+
|
15
|
+
Simply add the new column to an Excel/Open Office spreadsheet, or CSV file, and add the new
|
16
|
+
attribute or association name to the header row. Loader will attempt to find correct association and populate AR object with row data.
|
17
|
+
|
18
|
+
The Loader attempts to handle various human read-able forms of column names.
|
19
|
+
|
20
|
+
For example, given an association on the model called, product_properties, will successfully load
|
21
|
+
from columns with headings such as 'product_properties', 'Product Properties', 'ProductProperties' 'product properties' etc
|
22
|
+
|
23
|
+
For has_many associations, either multiple columns can be used or multiple values can be specified in a single column using suitable delimiters.
|
24
|
+
|
25
|
+
Complex associations/mappings, for example requiring complex lookups, can be handled by extending the loader engine.
|
26
|
+
|
27
|
+
Original focus was on support for the Open Source Spree e-commerce project, so includes specific loaders and rake tasks
|
28
|
+
for loading Spree Products, and associated data such as Product Variants, and Images.
|
29
|
+
|
30
|
+
## Installation
|
31
|
+
|
32
|
+
Add gem 'ar_loader' to your Gemfile/bundle, or install the latest gem as usual :
|
33
|
+
|
34
|
+
`gem install ar_loader`
|
35
|
+
|
36
|
+
To use :
|
37
|
+
|
38
|
+
gem 'ar_loader'
|
39
|
+
require 'ar_loader'
|
40
|
+
|
41
|
+
ArLoader::load_tasks
|
42
|
+
|
43
|
+
|
44
|
+
To pull the tasks in, add call in your Rakefile :
|
45
|
+
|
46
|
+
ArLoader::load_tasks
|
47
|
+
|
48
|
+
N.B - To use the Excel loader, OLE and Excel are NOT required, however
|
49
|
+
JRuby is required, since it uses Java's Apache POI under the hood to process .xls files.
|
50
|
+
|
51
|
+
To use in a mixed Ruby setup, you can use a guard something like :
|
52
|
+
|
53
|
+
if(RUBY_PLATFORM =~ /java/)
|
54
|
+
gem 'activerecord-jdbcmysql-adapter'
|
55
|
+
else
|
56
|
+
gem 'mysql'
|
57
|
+
end
|
58
|
+
|
59
|
+
## Example Spreadsheet
|
60
|
+
|
61
|
+
A number of example Spreadsheets with headers and comments, can be found in the spec/fixtures directory.
|
62
|
+
|
63
|
+
|
64
|
+
## Features
|
65
|
+
|
66
|
+
- *Direct Excel file support*
|
67
|
+
|
68
|
+
Includes a wrapper around MS Excel File format, via Apache POI, which
|
69
|
+
enables Products to be loaded directly from Excel files (Excel does not need to be installed) via JRuby.
|
70
|
+
No need to save to CSV first.
|
71
|
+
|
72
|
+
The java jars e.g - 'poi-3.6.jar' - are included.
|
73
|
+
|
74
|
+
- *Semi-Smart Name Lookup*
|
75
|
+
|
76
|
+
Includes helper classes that find and store details of all possible associations on an AR class.
|
77
|
+
Given a user supplied name, attempts to find the requested association.
|
78
|
+
|
79
|
+
Example usage, load from a file or spreadsheet where the column names are only
|
80
|
+
an approximation of the actual associations, so given 'Product Properties' heading,
|
81
|
+
finds real association 'product_properties' to send or call on the AR object
|
82
|
+
|
83
|
+
- *Associations*
|
84
|
+
|
85
|
+
Can handle 'belongs_to, 'has_many' and 'has_one' associations, including assignment of multiple objects
|
86
|
+
via either multiple columns, or via specially delimited entry in a single (column). See Details section.
|
87
|
+
|
88
|
+
|
89
|
+
- *Rake Tasks*
|
90
|
+
|
91
|
+
High level Rake tasks are provided, only required to supply model class, and file location :
|
92
|
+
|
93
|
+
jruby -S rake ar_loader:excel model=MusicTrack input=MyTrackListing.xls
|
94
|
+
|
95
|
+
|
96
|
+
- *Spree Rake Tasks*
|
97
|
+
|
98
|
+
Specific Rake tasks are also provided for Spree loading - currently supports Product with associations,
|
99
|
+
and Image loading.
|
100
|
+
|
101
|
+
jruby -S rake ar_loader:spree:products input=C:\MyProducts.xls
|
102
|
+
|
103
|
+
|
104
|
+
**Product loading from Excel files specifically requires JRuby (But not Excel or OLE)**.
|
105
|
+
|
106
|
+
|
107
|
+
- *Seamless Spree Image loading can be achieved by ensuring SKU or class Name features in Image filename.
|
108
|
+
|
109
|
+
Lookup is performed either via the SKU being prepended to the image name, or by the image name being equal to the **name attribute** of the klass in question.
|
110
|
+
|
111
|
+
Images can be attached to any class defined with a suitable association. The class to use can be configured in rake task via
|
112
|
+
parameter klass=Xyz.
|
113
|
+
|
114
|
+
In the Spree tasks, this defaults to Product, so attempts to attach Image to a Product via Product SKU or Name.
|
115
|
+
|
116
|
+
Image loading **does not** specifically require JRuby
|
117
|
+
|
118
|
+
A report is generated in the current working directory detailing any Images in the paths that could not be matched with a Product.
|
119
|
+
|
120
|
+
rake ar_loader:spree:images input=C:\images\product_images skip_if_no_assoc=true
|
121
|
+
|
122
|
+
rake ar_loader:spree:images input=C:\images\taxon_icons skip_if_no_assoc=true klass=Taxon
|
123
|
+
|
124
|
+
## Example Wrapper Tasks for Spree Site Extension
|
125
|
+
|
126
|
+
These tasks show how to write your own high level wrapper task, that will seed the database from multiple spreedsheets.
|
127
|
+
|
128
|
+
The images in this example have been named with the SKU present in name (separated by whitespace) e.g "PRINT_001 Stonehenge.jpg"
|
129
|
+
|
130
|
+
A report is generated in the current working directory detailing any Images in the paths that could not be matched with a Product.
|
131
|
+
|
132
|
+
require 'ar_loader'
|
133
|
+
|
134
|
+
namespace :mysite do
|
135
|
+
|
136
|
+
desc "Load Products for site"
|
137
|
+
task :load, :needs => [:environment] do |t, args|
|
138
|
+
|
139
|
+
[ "vendor/extensions/site/db/seed/Paintings.xls",
|
140
|
+
"vendor/extensions/site/db/seed/Drawings.xls"
|
141
|
+
].each do |x|
|
142
|
+
Rake::Task['ar_loader:spree:products'].execute(
|
143
|
+
:input => x,
|
144
|
+
:verbose => true,
|
145
|
+
:sku_prefix => ""
|
146
|
+
)
|
147
|
+
end
|
148
|
+
end
|
149
|
+
|
150
|
+
desc "Load Images for site based on SKU"
|
151
|
+
task :load_images, :clean, :dummy, :needs => [:environment] do |t, args|
|
152
|
+
|
153
|
+
if(args[:clean])
|
154
|
+
Image.delete_all
|
155
|
+
FileUtils.rm_rf( "public/assests/products" )
|
156
|
+
end
|
157
|
+
|
158
|
+
["01_paintings_jpegs", "02_drawings_jpegs"].each do |x|
|
159
|
+
|
160
|
+
# image names start with associated Product SKU,
|
161
|
+
# skip rather then exit if no matching product found
|
162
|
+
|
163
|
+
Rake::Task['autotelik:image_load'].execute(
|
164
|
+
:input => "/my_site_load_info//#{x}",
|
165
|
+
:dummy => args[:dummy],
|
166
|
+
:verbose => false, :sku => true, :skip_if_no_assoc => true
|
167
|
+
)
|
168
|
+
end
|
169
|
+
end
|
170
|
+
|
171
|
+
## Details
|
172
|
+
|
173
|
+
### Associations
|
174
|
+
|
175
|
+
To perform a lookup for an associated model, the primary column(s) must be supplied, along with required select values for those columns.
|
176
|
+
|
177
|
+
A single association column can contain multiple name/value sets, in string form :
|
178
|
+
|
179
|
+
column:lookup_key_1, lookup_key_2,...
|
180
|
+
|
181
|
+
So if our Project model has many Categories, we can supply a Category list, which is keyed on the column 'reference' with :
|
182
|
+
|
183
|
+
|Categories|
|
184
|
+
reference:category_001,category_002
|
185
|
+
|
186
|
+
During loading, a call to find_all_by_reference will be made, picking up the 2 categories with matching references,
|
187
|
+
and our Project model will contain those two i.e project.categories = [category_002,category_003]
|
188
|
+
|
189
|
+
## Spree Suppprt
|
190
|
+
|
191
|
+
### OptionTypes & Variants
|
192
|
+
|
193
|
+
When loaded with the Spree specific tasks, spree specific over rides are supported, such as direct s
|
194
|
+
support for OptionTypes with values
|
195
|
+
|
196
|
+
Any 'Option Types' columns can contain the OptionType to associate with the Product, plus a selection of
|
197
|
+
appropriate OptionValues to go with that Type.
|
198
|
+
|
199
|
+
For example, in a single column/row we could supply 2 OptionTypes (named, size & colour), with a selection values
|
200
|
+
(such as small, medium etc)
|
201
|
+
|
202
|
+
'Option Types'
|
203
|
+
size:small,medium,large|colour:red,white
|
204
|
+
|
205
|
+
If no such OptionType exists, e.g size, then a new one is created with the supplied name.
|
206
|
+
|
207
|
+
Next the OptionValues are also parsed, again if no such OptionValue exists, e.g small, then a new one is created with the supplied name.
|
208
|
+
|
209
|
+
Lastly a Variant is created on each OptionValue, with price and availaable dates being copied from Master.
|
210
|
+
Currently a unique SKU is created by adding an index to the master's sku.
|
211
|
+
|
212
|
+
TODO - Enable a hash of attributes to be supplied in association columns to enable more control over creation of associated objects.
|
213
|
+
|
214
|
+
### Properties
|
215
|
+
|
216
|
+
The properties to associate with this product.
|
217
|
+
Properties are for small snippets of text, shared across many products,
|
218
|
+
and are for display purposes only.
|
219
|
+
|
220
|
+
An optional display value can be supplied to supplement the displayed text.
|
221
|
+
|
222
|
+
As for all associations can contain multiple name/value sets in default form :
|
223
|
+
|
224
|
+
Property:display_value|Property:display_value
|
225
|
+
|
226
|
+
Example - No values :
|
227
|
+
manufacturer|standard
|
228
|
+
|
229
|
+
Example - Display values :
|
230
|
+
manufacturer:somebody else plc|standard:ISOBlah21
|
231
|
+
|
232
|
+
## TODO
|
233
|
+
|
234
|
+
- Directly support csv,
|
235
|
+
when JRuby and/or Excel not available.
|
236
|
+
|
237
|
+
- Smart sorting of column processing order ....
|
238
|
+
|
239
|
+
Does not currently ensure mandatory columns (for valid?) processed first.
|
240
|
+
Since Product needs saving before associations can be processed, user currently
|
241
|
+
needs to ensure SKU, name, price columns are among first columns
|
242
|
+
|
243
|
+
## License
|
244
|
+
|
245
|
+
Copyright:: (c) Autotelik Media Ltd 2011
|
246
|
+
|
247
|
+
Author :: Tom Statter
|
248
|
+
|
249
|
+
Date :: Feb 2011
|
250
|
+
|
251
|
+
The MIT License
|
252
|
+
|
253
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
254
|
+
of this software and associated documentation files (the "Software"), to deal
|
255
|
+
in the Software without restriction, including without limitation the rights
|
256
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
257
|
+
copies of the Software, and to permit persons to whom the Software is
|
258
|
+
furnished to do so, subject to the following conditions:
|
259
|
+
|
260
|
+
The above copyright notice and this permission notice shall be included in
|
261
|
+
all copies or substantial portions of the Software.
|
262
|
+
|
263
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
264
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
265
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
266
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
267
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
268
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
222
269
|
THE SOFTWARE.
|