ar_loader 0.0.9 → 1.0.0.0
Sign up to get free protection for your applications and to get access to all the features.
- data/README.markdown +2 -243
- data/Rakefile +1 -1
- data/lib/VERSION +1 -1
- data/lib/generators/excel_generator.rb +27 -19
- data/lib/helpers/jruby/jexcel_file.rb +1 -61
- data/lib/java/poi-3.2-FINAL-20081019.jar +0 -0
- data/spec/excel_generator_spec.rb +6 -26
- data/spec/fixtures/simple_export_spec.xls +0 -0
- data/tasks/excel_generator.rake +0 -31
- metadata +8 -11
- data/spec/fixtures/results/simple_export_spec.xls +0 -0
- data/spec/fixtures/simple_template_spec.xls +0 -0
- data/spec/logs/test.log +0 -5369
data/README.markdown
CHANGED
@@ -1,247 +1,6 @@
|
|
1
1
|
## AR Loader
|
2
2
|
|
3
|
-
|
4
|
-
including csv files and .xls files (Excel Spreadsheets)
|
5
|
-
|
6
|
-
Simplifies the specification and loading of data from such files into any active record supported database.
|
7
|
-
|
8
|
-
Aims to generically and seamlessly, handle loading an active record model's attributes and it's associations,
|
9
|
-
based on reflection against the supplied model.
|
10
|
-
|
11
|
-
So rather than hard coded mappings, uses the file's column headings to map data to a model's attributes and associations.
|
12
|
-
|
13
|
-
This makes loaders extendable via column/file data rather than requiring new Ruby coding.
|
14
|
-
|
15
|
-
Simply add the new column to an Excel/Open Office spreadsheet, or CSV file, and add the new
|
16
|
-
attribute or association name to the header row. Loader will attempt to find correct association and populate AR object with row data.
|
17
|
-
|
18
|
-
The Loader attempts to handle various human read-able forms of column names.
|
19
|
-
|
20
|
-
For example, given an association on the model called, product_properties, will successfully load
|
21
|
-
from columns with headings such as 'product_properties', 'Product Properties', 'ProductProperties' 'product properties' etc
|
22
|
-
|
23
|
-
For has_many associations, either multiple columns can be used or multiple values can be specified in a single column using suitable delimiters.
|
24
|
-
|
25
|
-
Complex associations/mappings, for example requiring complex lookups, can be handled by extending the loader engine.
|
26
|
-
|
27
|
-
Original focus was on support for the Open Source Spree e-commerce project, so includes specific loaders and rake tasks
|
28
|
-
for loading Spree Products, and associated data such as Product Variants, and Images.
|
29
|
-
|
30
|
-
## Installation
|
31
|
-
|
32
|
-
Add gem 'ar_loader' to your Gemfile/bundle, or install the latest gem as usual :
|
33
|
-
|
34
|
-
`gem install ar_loader`
|
35
|
-
|
36
|
-
To use :
|
37
|
-
|
38
|
-
gem 'ar_loader'
|
39
|
-
require 'ar_loader'
|
40
|
-
|
41
|
-
ArLoader::load_tasks
|
42
|
-
|
43
|
-
|
44
|
-
To pull the tasks in, add call in your Rakefile :
|
45
|
-
|
46
|
-
ArLoader::load_tasks
|
47
|
-
|
48
|
-
N.B - To use the Excel loader, OLE and Excel are NOT required, however
|
49
|
-
JRuby is required, since it uses Java's Apache POI under the hood to process .xls files.
|
50
|
-
|
51
|
-
To use in a mixed Ruby setup, you can use a guard something like :
|
52
|
-
|
53
|
-
if(RUBY_PLATFORM =~ /java/)
|
54
|
-
gem 'activerecord-jdbcmysql-adapter'
|
55
|
-
else
|
56
|
-
gem 'mysql'
|
57
|
-
end
|
58
|
-
|
59
|
-
## Example Spreadsheet
|
60
|
-
|
61
|
-
A number of example Spreadsheets with headers and comments, can be found in the spec/fixtures directory.
|
62
|
-
|
63
|
-
|
64
|
-
## Features
|
65
|
-
|
66
|
-
- *Direct Excel file support*
|
67
|
-
|
68
|
-
Includes a wrapper around MS Excel File format, via Apache POI, which
|
69
|
-
enables Products to be loaded directly from Excel files (Excel does not need to be installed) via JRuby.
|
70
|
-
No need to save to CSV first.
|
71
|
-
|
72
|
-
The java jars e.g - 'poi-3.6.jar' - are included.
|
73
|
-
|
74
|
-
- *Semi-Smart Name Lookup*
|
75
|
-
|
76
|
-
Includes helper classes that find and store details of all possible associations on an AR class.
|
77
|
-
Given a user supplied name, attempts to find the requested association.
|
78
|
-
|
79
|
-
Example usage, load from a file or spreadsheet where the column names are only
|
80
|
-
an approximation of the actual associations, so given 'Product Properties' heading,
|
81
|
-
finds real association 'product_properties' to send or call on the AR object
|
82
|
-
|
83
|
-
- *Associations*
|
84
|
-
|
85
|
-
Can handle 'belongs_to, 'has_many' and 'has_one' associations, including assignment of multiple objects
|
86
|
-
via either multiple columns, or via specially delimited entry in a single (column). See Details section.
|
87
|
-
|
88
|
-
|
89
|
-
- *Rake Tasks*
|
90
|
-
|
91
|
-
High level Rake tasks are provided, only required to supply model class, and file location :
|
92
|
-
|
93
|
-
jruby -S rake ar_loader:excel model=MusicTrack input=MyTrackListing.xls
|
94
|
-
|
95
|
-
|
96
|
-
- *Spree Rake Tasks*
|
97
|
-
|
98
|
-
Specific Rake tasks are also provided for Spree loading - currently supports Product with associations,
|
99
|
-
and Image loading.
|
100
|
-
|
101
|
-
jruby -S rake ar_loader:spree:products input=C:\MyProducts.xls
|
102
|
-
|
103
|
-
|
104
|
-
**Product loading from Excel files specifically requires JRuby (But not Excel or OLE)**.
|
105
|
-
|
106
|
-
|
107
|
-
- *Seamless Spree Image loading can be achieved by ensuring SKU or class Name features in Image filename.
|
108
|
-
|
109
|
-
Lookup is performed either via the SKU being prepended to the image name, or by the image name being equal to the **name attribute** of the klass in question.
|
110
|
-
|
111
|
-
Images can be attached to any class defined with a suitable association. The class to use can be configured in rake task via
|
112
|
-
parameter klass=Xyz.
|
113
|
-
|
114
|
-
In the Spree tasks, this defaults to Product, so attempts to attach Image to a Product via Product SKU or Name.
|
115
|
-
|
116
|
-
Image loading **does not** specifically require JRuby
|
117
|
-
|
118
|
-
A report is generated in the current working directory detailing any Images in the paths that could not be matched with a Product.
|
119
|
-
|
120
|
-
rake ar_loader:spree:images input=C:\images\product_images skip_if_no_assoc=true
|
121
|
-
|
122
|
-
rake ar_loader:spree:images input=C:\images\taxon_icons skip_if_no_assoc=true klass=Taxon
|
123
|
-
|
124
|
-
## Example Wrapper Tasks for Spree Site Extension
|
125
|
-
|
126
|
-
These tasks show how to write your own high level wrapper task, that will seed the database from multiple spreedsheets.
|
127
|
-
|
128
|
-
The images in this example have been named with the SKU present in name (separated by whitespace) e.g "PRINT_001 Stonehenge.jpg"
|
129
|
-
|
130
|
-
A report is generated in the current working directory detailing any Images in the paths that could not be matched with a Product.
|
131
|
-
|
132
|
-
require 'ar_loader'
|
133
|
-
|
134
|
-
namespace :mysite do
|
135
|
-
|
136
|
-
desc "Load Products for site"
|
137
|
-
task :load, :needs => [:environment] do |t, args|
|
138
|
-
|
139
|
-
[ "vendor/extensions/site/db/seed/Paintings.xls",
|
140
|
-
"vendor/extensions/site/db/seed/Drawings.xls"
|
141
|
-
].each do |x|
|
142
|
-
Rake::Task['ar_loader:spree:products'].execute(
|
143
|
-
:input => x,
|
144
|
-
:verbose => true,
|
145
|
-
:sku_prefix => ""
|
146
|
-
)
|
147
|
-
end
|
148
|
-
end
|
149
|
-
|
150
|
-
desc "Load Images for site based on SKU"
|
151
|
-
task :load_images, :clean, :dummy, :needs => [:environment] do |t, args|
|
152
|
-
|
153
|
-
if(args[:clean])
|
154
|
-
Image.delete_all
|
155
|
-
FileUtils.rm_rf( "public/assests/products" )
|
156
|
-
end
|
157
|
-
|
158
|
-
["01_paintings_jpegs", "02_drawings_jpegs"].each do |x|
|
159
|
-
|
160
|
-
# image names start with associated Product SKU,
|
161
|
-
# skip rather then exit if no matching product found
|
162
|
-
|
163
|
-
Rake::Task['autotelik:image_load'].execute(
|
164
|
-
:input => "/my_site_load_info//#{x}",
|
165
|
-
:dummy => args[:dummy],
|
166
|
-
:verbose => false, :sku => true, :skip_if_no_assoc => true
|
167
|
-
)
|
168
|
-
end
|
169
|
-
end
|
170
|
-
|
171
|
-
## Details
|
172
|
-
|
173
|
-
### Associations
|
174
|
-
|
175
|
-
To perform a lookup for an associated model, the primary column(s) must be supplied, along with required select values for those columns.
|
176
|
-
|
177
|
-
A single association column can contain multiple name/value sets, in string form :
|
178
|
-
|
179
|
-
column:lookup_key_1, lookup_key_2,...
|
180
|
-
|
181
|
-
So if our Project model has many Categories, we can supply a Category list, which is keyed on the column 'reference' with :
|
182
|
-
|
183
|
-
|Categories|
|
184
|
-
|
185
|
-
reference:category_001,category_002
|
186
|
-
|
187
|
-
During loading, a call to find_all_by_reference will be made, picking up the 2 categories with matching references,
|
188
|
-
and our Project model will contain those two i.e project.categories = [category_002,category_003]
|
189
|
-
|
190
|
-
## Spree Suppprt
|
191
|
-
|
192
|
-
### OptionTypes & Variants
|
193
|
-
|
194
|
-
When loaded with the Spree specific tasks, spree specific over rides are supported, such as direct s
|
195
|
-
support for OptionTypes with values
|
196
|
-
|
197
|
-
Any 'Option Types' columns can contain the OptionType to associate with the Product, plus a selection of
|
198
|
-
appropriate OptionValues to go with that Type.
|
199
|
-
|
200
|
-
For example, in a single column/row we could supply 2 OptionTypes (named, size & colour), with a selection values
|
201
|
-
(such as small, medium etc)
|
202
|
-
|
203
|
-
'Option Types'
|
204
|
-
size:small,medium,large|colour:red,white
|
205
|
-
|
206
|
-
If no such OptionType exists, e.g size, then a new one is created with the supplied name.
|
207
|
-
|
208
|
-
Next the OptionValues are also parsed, again if no such OptionValue exists, e.g small, then a new one is created with the supplied name.
|
209
|
-
|
210
|
-
Lastly a Variant is created on each OptionValue, with price and availaable dates being copied from Master.
|
211
|
-
Currently a unique SKU is created by adding an index to the master's sku.
|
212
|
-
|
213
|
-
TODO - Enable a hash of attributes to be supplied in association columns to enable more control over creation of associated objects.
|
214
|
-
|
215
|
-
### Properties
|
216
|
-
|
217
|
-
The properties to associate with this product.
|
218
|
-
Properties are for small snippets of text, shared across many products,
|
219
|
-
and are for display purposes only.
|
220
|
-
|
221
|
-
An optional display value can be supplied to supplement the displayed text.
|
222
|
-
|
223
|
-
As for all associations can contain multiple name/value sets in default form :
|
224
|
-
|
225
|
-
Property:display_value|Property:display_value
|
226
|
-
|
227
|
-
Example - No values :
|
228
|
-
manufacturer|standard
|
229
|
-
|
230
|
-
Example - Display values :
|
231
|
-
manufacturer:somebody else plc|standard:ISOBlah21
|
232
|
-
|
233
|
-
## TODO
|
234
|
-
|
235
|
-
- Directly support csv,
|
236
|
-
when JRuby and/or Excel not available.
|
237
|
-
|
238
|
-
- Smart sorting of column processing order ....
|
239
|
-
|
240
|
-
Does not currently ensure mandatory columns (for valid?) processed first.
|
241
|
-
Since Product needs saving before associations can be processed, user currently
|
242
|
-
needs to ensure SKU, name, price columns are among first columns
|
243
|
-
|
244
|
-
## License
|
3
|
+
DEFUNCT - Please see replacement project https://github.com/autotelik/datashift
|
245
4
|
|
246
5
|
Copyright:: (c) Autotelik Media Ltd 2011
|
247
6
|
|
@@ -267,4 +26,4 @@ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
267
26
|
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
268
27
|
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
269
28
|
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
270
|
-
THE SOFTWARE.
|
29
|
+
THE SOFTWARE.
|
data/Rakefile
CHANGED
@@ -29,7 +29,7 @@ spec = Gem::Specification.new do |s|
|
|
29
29
|
s.has_rdoc = true
|
30
30
|
s.extra_rdoc_files = ['README.markdown', 'LICENSE']
|
31
31
|
s.summary = 'File based loader for Active Record models'
|
32
|
-
s.description = '
|
32
|
+
s.description = 'DEFUNCT - Please see https://github.com/autotelik/datashift instead'
|
33
33
|
s.author = 'thomas statter'
|
34
34
|
s.email = 'rubygems@autotelik.co.uk'
|
35
35
|
s.date = DateTime.now.strftime("%Y-%m-%d")
|
data/lib/VERSION
CHANGED
@@ -1 +1 @@
|
|
1
|
-
0.0.
|
1
|
+
1.0.0.0
|
@@ -16,21 +16,18 @@ if(Guards::jruby?)
|
|
16
16
|
|
17
17
|
class ExcelGenerator < GeneratorBase
|
18
18
|
|
19
|
-
attr_accessor :excel
|
19
|
+
attr_accessor :excel
|
20
20
|
|
21
21
|
|
22
|
-
def initialize
|
22
|
+
def initialize
|
23
23
|
@excel = nil
|
24
|
-
@filename = filename
|
25
24
|
end
|
26
25
|
|
27
26
|
|
28
27
|
# Create an Excel file representing supplied Model
|
29
28
|
|
30
|
-
def generate(model,
|
31
|
-
|
32
|
-
@filename = options[:filename] if options[:filename]
|
33
|
-
|
29
|
+
def generate(model, filename)
|
30
|
+
|
34
31
|
@excel = JExcelFile.new()
|
35
32
|
|
36
33
|
sheet = @excel.create_sheet( model.name )
|
@@ -53,22 +50,33 @@ if(Guards::jruby?)
|
|
53
50
|
end
|
54
51
|
|
55
52
|
|
56
|
-
|
57
|
-
|
58
|
-
def export(items, options = {})
|
59
|
-
|
60
|
-
@filename = options[:filename] if options[:filename]
|
61
|
-
|
62
|
-
@excel = JExcelFile.new()
|
53
|
+
def to_xls(items=[])
|
63
54
|
|
64
|
-
|
55
|
+
# value rows
|
56
|
+
row_index = 1
|
57
|
+
items.each do |item|
|
58
|
+
row = sheet.createRow(row_index);
|
65
59
|
|
66
|
-
|
60
|
+
cell_index = 0
|
61
|
+
item.class.columns.each do |column|
|
62
|
+
cell = row.createCell(cell_index)
|
63
|
+
if column.sql_type =~ /date/ then
|
64
|
+
millis = item.send(column.name).to_f * 1000
|
65
|
+
cell.setCellValue(Date.new(millis))
|
66
|
+
cell.setCellStyle(dateStyle);
|
67
|
+
elsif column.sql_type =~ /int/ then
|
68
|
+
cell.setCellValue(item.send(column.name).to_i)
|
69
|
+
else
|
70
|
+
value = item.send(column.name)
|
71
|
+
cell.setCellValue(item.send(column.name)) unless value.nil?
|
72
|
+
end
|
73
|
+
cell_index += 1
|
74
|
+
end
|
75
|
+
row_index += 1
|
76
|
+
end
|
77
|
+
@excel.to_s
|
67
78
|
end
|
68
|
-
|
69
79
|
end
|
70
80
|
|
71
81
|
end
|
72
|
-
else
|
73
|
-
raise "BAD RUBY : PLEASE USE JRuby - Sorry JExcelFile requires JAVA via JRuby"
|
74
82
|
end # jruby
|
@@ -42,10 +42,6 @@ if(Guards::jruby?)
|
|
42
42
|
MAX_COLUMNS = 256.freeze
|
43
43
|
MAX_ROWS = 65536.freeze
|
44
44
|
|
45
|
-
def self.date_format
|
46
|
-
HSSFDataFormat.getBuiltinFormat("m/d/yy h:mm")
|
47
|
-
end
|
48
|
-
|
49
45
|
# The HSSFWorkbook uses 0 based indexes
|
50
46
|
|
51
47
|
def initialize()
|
@@ -62,7 +58,6 @@ if(Guards::jruby?)
|
|
62
58
|
|
63
59
|
# TOFIX - how do we know which sheet we are creating so we can set index @current_sheet
|
64
60
|
def create_sheet(sheet_name)
|
65
|
-
@current_sheet = 0
|
66
61
|
@book = HSSFWorkbook.new()
|
67
62
|
@sheet = @book.createSheet(sheet_name.gsub(" ", ''))
|
68
63
|
date_style = @book.createCellStyle()
|
@@ -122,60 +117,6 @@ if(Guards::jruby?)
|
|
122
117
|
end
|
123
118
|
|
124
119
|
|
125
|
-
def set_headers(headers)
|
126
|
-
|
127
|
-
return if headers.empty?
|
128
|
-
|
129
|
-
create_sheet( headers.first.class.name ) unless(@sheet)
|
130
|
-
|
131
|
-
row = sheet.createRow(0)
|
132
|
-
|
133
|
-
headers.each_with_index.each do |column, i|
|
134
|
-
row.createCell(i).setCellValue(column.name.to_s)
|
135
|
-
end
|
136
|
-
end
|
137
|
-
|
138
|
-
|
139
|
-
def to_xls(items)
|
140
|
-
return if items.empty?
|
141
|
-
|
142
|
-
create_sheet( items.first.class.name ) unless(@sheet)
|
143
|
-
|
144
|
-
set_headers( items.first.class.columns )
|
145
|
-
|
146
|
-
date_style = @book.createCellStyle
|
147
|
-
date_style.setDataFormat( JExcelFile::date_format )
|
148
|
-
|
149
|
-
# value rows
|
150
|
-
row_index = 1
|
151
|
-
items.each do |item|
|
152
|
-
row = sheet.createRow(row_index);
|
153
|
-
|
154
|
-
cell_index = 0
|
155
|
-
item.class.columns.each do |column|
|
156
|
-
cell = row.createCell(cell_index)
|
157
|
-
|
158
|
-
if column.sql_type =~ /date/ then
|
159
|
-
|
160
|
-
millis = item.send(column.name).to_f * 1000
|
161
|
-
cell.setCellValue(Date.new(millis))
|
162
|
-
|
163
|
-
cell.setCellStyle( date_style );
|
164
|
-
elsif column.sql_type =~ /int/ then
|
165
|
-
cell.setCellValue(item.send(column.name).to_i)
|
166
|
-
|
167
|
-
else
|
168
|
-
value = item.send(column.name)
|
169
|
-
cell.setCellValue( value.to_s ) unless value.nil?
|
170
|
-
end
|
171
|
-
|
172
|
-
cell_index += 1
|
173
|
-
end
|
174
|
-
row_index += 1
|
175
|
-
end
|
176
|
-
@excel.to_s
|
177
|
-
end
|
178
|
-
|
179
120
|
# The internal representation of a Excel File
|
180
121
|
|
181
122
|
def to_s
|
@@ -186,6 +127,5 @@ if(Guards::jruby?)
|
|
186
127
|
end
|
187
128
|
|
188
129
|
end
|
189
|
-
|
190
|
-
raise "Sorry can only access JExcelFile using JRuby"
|
130
|
+
|
191
131
|
end
|
Binary file
|
@@ -31,42 +31,22 @@ describe 'Excel Generator' do
|
|
31
31
|
end
|
32
32
|
|
33
33
|
it "should be able to create a new excel generator" do
|
34
|
-
generator = ExcelGenerator.new(
|
34
|
+
generator = ExcelGenerator.new( )
|
35
35
|
end
|
36
36
|
|
37
|
-
it "should
|
37
|
+
it "should export a simple model to .xls spreedsheet" do
|
38
38
|
|
39
|
-
expect= $fixture_path + '/
|
39
|
+
expect= $fixture_path + '/simple_export_spec.xls'
|
40
40
|
|
41
41
|
begin FileUtils.rm(expect); rescue; end
|
42
42
|
|
43
|
-
gen = ExcelGenerator.new
|
43
|
+
gen = ExcelGenerator.new
|
44
44
|
|
45
|
-
gen.generate(@klazz)
|
45
|
+
gen.generate(@klazz, expect)
|
46
46
|
|
47
47
|
File.exists?(expect).should be_true
|
48
48
|
|
49
49
|
end
|
50
50
|
|
51
|
-
|
52
|
-
it "should export a simple model to .xls spreedsheet" do
|
53
|
-
|
54
|
-
Project.create( :value_as_string => 'Value as Text', :value_as_boolean => true, :value_as_double => 75.672)
|
55
|
-
#001 Demo string blah blah 2011-02-14 1.00 320.00
|
56
|
-
|
57
|
-
|
58
|
-
expect= $fixture_path + '/results/simple_export_spec.xls'
|
59
|
-
|
60
|
-
begin FileUtils.rm(expect); rescue; end
|
61
|
-
|
62
|
-
gen = ExcelGenerator.new(expect)
|
63
|
-
|
64
|
-
items = Project.all
|
65
|
-
|
66
|
-
gen.export(items)
|
67
|
-
|
68
|
-
File.exists?(expect).should be_true
|
69
|
-
|
70
|
-
end
|
71
|
-
|
51
|
+
|
72
52
|
end
|