ar_loader 0.0.9 → 1.0.0.0

data/README.markdown

@@ -1,247 +1,6 @@
 ## AR Loader
 
-General active record loader for populating database with seed data from various sources,
-including csv files and .xls files (Excel Spreadsheets)
-
-Simplifies the specification and loading of data from such files into any active record supported database.
-
-Aims to generically and seamlessly, handle loading an active record model's attributes and it's associations,
-based on reflection against the supplied model.
-
-So rather than hard coded mappings, uses the file's column headings to map data to a model's attributes and associations.
-
-This makes loaders extendable via column/file data rather than requiring new Ruby coding.
-
-Simply add the new column to an Excel/Open Office spreadsheet, or CSV file, and add the new
-attribute or association name to the header row. Loader will attempt to find correct association and populate AR object with row data.
-
-The Loader attempts to handle various human read-able forms of column names.
-
-For example, given an association on the model called, product_properties, will successfully load
-from columns with headings such as 'product_properties', 'Product Properties', 'ProductProperties' 'product properties' etc
-
-For has_many associations, either multiple columns can be used or multiple values can be specified in a single column using suitable delimiters.
-
-Complex associations/mappings, for example requiring complex lookups, can be handled by extending the loader engine.
-
-Original focus was on support for the Open Source Spree e-commerce project, so includes specific loaders and rake tasks
-for loading Spree Products, and associated data such as Product Variants, and Images.
-
-## Installation
-
-Add gem 'ar_loader' to your Gemfile/bundle, or install the latest gem as usual :
-
-    `gem install ar_loader`
-
-To use :
-
-    gem 'ar_loader'
-    require 'ar_loader'
-
-    ArLoader::load_tasks
-
-
-To pull the tasks in, add call in your Rakefile :
-
-    ArLoader::load_tasks
-
-N.B - To use the Excel loader, OLE and Excel are NOT required, however
-JRuby is required, since it uses Java's Apache POI under the hood to process .xls files.
-
-To use in a mixed Ruby setup, you can use a guard something like :
-
-    if(RUBY_PLATFORM =~ /java/)
-         gem 'activerecord-jdbcmysql-adapter'
-    else
-        gem 'mysql'
-    end
-
-## Example Spreadsheet
-    
-  A number of example Spreadsheets with headers and comments, can be found in the spec/fixtures directory.
-
-
-## Features
-
-- *Direct Excel file support*
-
-  Includes a wrapper around MS Excel File format, via Apache POI, which
-  enables Products to be loaded directly from Excel files (Excel does not need to be installed) via JRuby.
-  No need to save to CSV first.
-
-  The java jars e.g - 'poi-3.6.jar' - are included.
-
-- *Semi-Smart Name Lookup*
-
-  Includes helper classes that find and store details of all possible associations on an AR class.
-  Given a user supplied name, attempts to find the requested association.
-
-  Example usage, load from a file or spreadsheet where the column names are only
-  an approximation of the actual associations, so given 'Product Properties' heading,
-  finds real association 'product_properties' to send or call on the AR object
-
-- *Associations*
-
-  Can handle 'belongs_to, 'has_many' and 'has_one' associations, including assignment of multiple objects
-  via either multiple columns, or via specially delimited entry in a single (column). See Details section.
-
-
-- *Rake Tasks*
-
-  High level Rake tasks are provided, only required to supply model class, and file location :
-
-    jruby -S rake ar_loader:excel model=MusicTrack input=MyTrackListing.xls
-
-
-- *Spree Rake Tasks*
-
-  Specific Rake tasks are also provided for Spree loading - currently supports Product with associations,
-  and Image loading.
-
-    jruby -S rake ar_loader:spree:products input=C:\MyProducts.xls
-
-
-  **Product loading from Excel files specifically requires JRuby (But not Excel or OLE)**.
-
-
-- *Seamless Spree Image loading can be achieved by ensuring SKU or class Name features in Image filename.
-
-  Lookup is performed either via the SKU being prepended to the image name, or by the image name being equal to the **name attribute** of the klass in question.
-
-  Images can be attached to any class defined with a suitable association. The class to use can be configured in rake task via
-  parameter klass=Xyz.
-
-  In the Spree tasks, this defaults to Product, so attempts to attach Image to a Product via Product SKU or Name.
- 
-  Image loading **does not** specifically require JRuby
-
-  A report is generated in the current working directory detailing any Images in the paths that could not be matched with a Product.
-
-  rake ar_loader:spree:images input=C:\images\product_images skip_if_no_assoc=true
-
-  rake ar_loader:spree:images input=C:\images\taxon_icons skip_if_no_assoc=true klass=Taxon
-
-## Example Wrapper Tasks for Spree Site Extension
-
-These tasks show how to write your own high level wrapper task, that will seed the database from multiple spreedsheets.
-
-The images in this example have been named with the SKU present in name (separated by whitespace) e.g "PRINT_001 Stonehenge.jpg"
-
-A report is generated in the current working directory detailing any Images in the paths that could not be matched with a Product.
-
-    require 'ar_loader'
-
-    namespace :mysite do
-
-    desc "Load Products for site"
-    task :load, :needs => [:environment] do |t, args|
-
-      [ "vendor/extensions/site/db/seed/Paintings.xls",
-        "vendor/extensions/site/db/seed/Drawings.xls"
-      ].each do |x|
-        Rake::Task['ar_loader:spree:products'].execute(
-          :input => x,
-          :verbose => true,
-          :sku_prefix => ""
-        )
-      end
-    end
-
-    desc "Load Images for site based on SKU"
-    task :load_images, :clean, :dummy, :needs => [:environment] do |t, args|
-
-      if(args[:clean])
-        Image.delete_all
-        FileUtils.rm_rf( "public/assests/products" )
-      end
-
-      ["01_paintings_jpegs", "02_drawings_jpegs"].each do |x|
-
-        # image names start with associated Product SKU,
-        # skip rather then exit if no matching product found
-
-        Rake::Task['autotelik:image_load'].execute(
-          :input => "/my_site_load_info//#{x}",
-          :dummy => args[:dummy],
-          :verbose => false, :sku => true, :skip_if_no_assoc => true
-        )  
-      end
-    end
-
-## Details
-
-### Associations
-
-To perform a lookup for an associated model, the primary column(s) must be supplied, along with required select values for those columns.
-
-A single association column can contain multiple name/value sets, in string form :
-
-  column:lookup_key_1, lookup_key_2,...
-
-So if our Project model has many Categories, we can supply a Category list, which is keyed on the column 'reference' with :
-
-  |Categories|
-
-  reference:category_001,category_002
-
-During loading, a call to find_all_by_reference will be made, picking up the 2 categories with matching references,
- and our Project model will contain those two i.e project.categories = [category_002,category_003]
-
-## Spree Suppprt
-
-### OptionTypes & Variants
-
-When loaded with the Spree specific tasks, spree specific over rides are supported, such as direct s
-support for OptionTypes with values
-
-Any 'Option Types' columns can contain the OptionType to associate with the Product, plus a selection of
-appropriate OptionValues to go with that Type. 
-
-For example, in a single column/row we could supply 2 OptionTypes (named, size & colour), with a selection values
-(such as small, medium etc)
-
-    'Option Types'
-    size:small,medium,large|colour:red,white
-
-If no such OptionType exists, e.g size, then a new one is created with the supplied name.
-
-Next the OptionValues are also parsed, again if no such OptionValue exists, e.g small, then a new one is created with the supplied name.
-
-Lastly a Variant is created on each OptionValue, with price and availaable dates being copied from Master.
-Currently a unique SKU is created by adding an index to the master's sku.
-
-TODO - Enable a hash of attributes to be supplied in association columns to enable more control over creation of associated objects.
-
-### Properties
-
-The properties to associate with this product.
-Properties are for small snippets of text, shared across many products,
-and are for display purposes only.
-
-An optional display value can be supplied to supplement the displayed text.
-
-As for all associations can contain multiple name/value sets in default form :
-
-    Property:display_value|Property:display_value
-
-Example - No values :
-    manufacturer|standard
-
-Example - Display  values :
-    manufacturer:somebody else plc|standard:ISOBlah21
-
-## TODO
-
-  - Directly support csv,
-    when JRuby and/or Excel not available.
-
-  - Smart sorting of column processing order ....
-
-    Does not currently ensure mandatory columns (for valid?) processed first.
-    Since Product needs saving before associations can be processed, user currently
-    needs to ensure SKU, name, price columns are among first columns
-
-## License
+DEFUNCT - Please see replacement project https://github.com/autotelik/datashift
 
 Copyright:: (c) Autotelik Media Ltd 2011
 
@@ -267,4 +26,4 @@ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-THE SOFTWARE.
+THE SOFTWARE.

data/Rakefile

@@ -29,7 +29,7 @@ spec = Gem::Specification.new do |s|
   s.has_rdoc = true
   s.extra_rdoc_files = ['README.markdown', 'LICENSE']
   s.summary = 'File based loader for Active Record models'
-  s.description = 'A file based loader for Active Record models. Seed database directly from Excel/CSV. Includes rake support for Spree'
+  s.description = 'DEFUNCT - Please see https://github.com/autotelik/datashift instead'
   s.author = 'thomas statter'
   s.email = 'rubygems@autotelik.co.uk'
   s.date = DateTime.now.strftime("%Y-%m-%d")

data/lib/VERSION

@@ -1 +1 @@
-0.0.9
+1.0.0.0

data/lib/generators/excel_generator.rb

@@ -16,21 +16,18 @@ if(Guards::jruby?)
 
     class ExcelGenerator < GeneratorBase
 
-      attr_accessor :excel, :filename
+      attr_accessor :excel
   
 
-      def initialize(filename)
+      def initialize
         @excel = nil
-        @filename = filename
       end
 
   
       # Create an Excel file representing supplied Model
     
-      def generate(model, options = {})
-
-        @filename = options[:filename] if  options[:filename]
-
+      def generate(model, filename)
+   
         @excel = JExcelFile.new()
 
         sheet = @excel.create_sheet( model.name )
@@ -53,22 +50,33 @@ if(Guards::jruby?)
       end
 
   
-      # Create an Excel file representing supplied Model
-
-      def export(items, options = {})
-
-        @filename = options[:filename] if  options[:filename]
-
-        @excel = JExcelFile.new()
+      def to_xls(items=[])
 
-        @excel.to_xls(items)
+        # value rows
+        row_index = 1
+        items.each do |item|
+          row = sheet.createRow(row_index);
 
-        @excel.save( filename )
+          cell_index = 0
+          item.class.columns.each do |column|
+            cell = row.createCell(cell_index)
+            if column.sql_type =~ /date/ then
+              millis = item.send(column.name).to_f * 1000
+              cell.setCellValue(Date.new(millis))
+              cell.setCellStyle(dateStyle);
+            elsif column.sql_type =~ /int/ then
+              cell.setCellValue(item.send(column.name).to_i)
+            else
+              value = item.send(column.name)
+              cell.setCellValue(item.send(column.name)) unless value.nil?
+            end
+            cell_index += 1
+          end
+          row_index += 1
+        end
+        @excel.to_s
       end
-
     end
  
   end
-else
-  raise "BAD RUBY : PLEASE USE JRuby - Sorry JExcelFile requires JAVA via JRuby"
 end # jruby

data/lib/helpers/jruby/jexcel_file.rb

@@ -42,10 +42,6 @@ if(Guards::jruby?)
     MAX_COLUMNS = 256.freeze
     MAX_ROWS = 65536.freeze
 
-    def self.date_format
-      HSSFDataFormat.getBuiltinFormat("m/d/yy h:mm")
-    end
-
     # The HSSFWorkbook uses 0 based indexes
   
     def initialize()
@@ -62,7 +58,6 @@ if(Guards::jruby?)
 
     # TOFIX - how do we know which sheet we are creating so we can set index @current_sheet
     def create_sheet(sheet_name)
-      @current_sheet = 0
       @book = HSSFWorkbook.new()
       @sheet = @book.createSheet(sheet_name.gsub(" ", ''))
       date_style = @book.createCellStyle()
@@ -122,60 +117,6 @@ if(Guards::jruby?)
     end
 
 
-    def set_headers(headers)
-
-      return if headers.empty?
-
-      create_sheet( headers.first.class.name ) unless(@sheet)
-
-      row = sheet.createRow(0)
-
-      headers.each_with_index.each do |column, i|
-        row.createCell(i).setCellValue(column.name.to_s)
-      end
-    end
-
-
-    def to_xls(items)
-      return if items.empty?
-
-      create_sheet( items.first.class.name ) unless(@sheet)
-
-      set_headers( items.first.class.columns )
-
-      date_style = @book.createCellStyle
-      date_style.setDataFormat( JExcelFile::date_format )
-
-      # value rows
-      row_index = 1
-      items.each do |item|
-        row = sheet.createRow(row_index);
-
-        cell_index = 0
-        item.class.columns.each do |column|
-          cell = row.createCell(cell_index)
-          
-          if column.sql_type =~ /date/ then
-
-            millis = item.send(column.name).to_f * 1000
-            cell.setCellValue(Date.new(millis))
-
-            cell.setCellStyle( date_style );
-          elsif column.sql_type =~ /int/ then
-            cell.setCellValue(item.send(column.name).to_i)
-
-          else
-            value = item.send(column.name)
-            cell.setCellValue( value.to_s ) unless value.nil?
-          end
-
-          cell_index += 1
-        end
-        row_index += 1
-      end
-      @excel.to_s
-    end
-
     # The internal representation of a Excel File
   
     def to_s
@@ -186,6 +127,5 @@ if(Guards::jruby?)
     end
  
   end
-else
-  raise "Sorry can only access JExcelFile using JRuby"
+  
 end

data/spec/excel_generator_spec.rb

@@ -31,42 +31,22 @@ describe 'Excel Generator' do
   end
   
   it "should be able to create a new excel generator" do
-    generator = ExcelGenerator.new( 'dummy.xls' )
+    generator = ExcelGenerator.new( )
   end
   
-  it "should genrate template .xls file from model" do
+  it "should export a simple model to .xls spreedsheet" do
 
-    expect= $fixture_path + '/simple_template_spec.xls'
+    expect= $fixture_path + '/simple_export_spec.xls'
 
     begin FileUtils.rm(expect); rescue; end
   
-    gen = ExcelGenerator.new( expect )
+    gen = ExcelGenerator.new
     
-    gen.generate(@klazz)
+    gen.generate(@klazz, expect)
  
     File.exists?(expect).should be_true
 
   end
 
-
-   it "should export a simple model to .xls spreedsheet" do
-
-    Project.create( :value_as_string	=> 'Value as Text', :value_as_boolean => true,	:value_as_double => 75.672)
-#001 Demo string	blah blah	2011-02-14	1.00	320.00
-
-
-    expect= $fixture_path + '/results/simple_export_spec.xls'
-
-    begin FileUtils.rm(expect); rescue; end
-
-    gen = ExcelGenerator.new(expect)
-
-    items = Project.all
-
-    gen.export(items)
-
-    File.exists?(expect).should be_true
-
-  end
-
+  
 end