ar_loader 0.0.4

data/LICENSE

@@ -0,0 +1,9 @@
+== ARLoader
+
+Copyright tom statter @ www.autotelik.co.uk 2011
+
+Not sure what is appropriate LICENSE
+
+Free to use in any project, commercial or otherwise as long as copyright not removed ??
+
+MIT ?

data/README.markdown

@@ -0,0 +1,211 @@
+# AR Loader
+
+General Active Record Loader with current focus on support for Spree.
+
+Maps column headings to attributes and associations.
+
+Fully extendable via spreadsheet headings - simply add new column to Excel with
+attribute or association name, and loader will attempt to
+find correct association and populate AR object with row data.
+
+Can handle human read-able forms of column names. For example, given an association on AR model called,
+product_properties, will map from column headings such as 'product_properties',
+'Product Properties', 'product properties'  etc
+
+## Installation
+
+Add gem instruction to your Gemfile. To use the Excel loader, JRuby is required, so to use in a mixed setup
+of JRuby and deployed to other Rubies, use the following guard.
+
+    if(RUBY_PLATFORM =~ /java/)
+         gem 'activerecord-jdbcmysql-adapter'
+    else
+        gem 'mysql'
+    end
+
+Currently not tested AR usage outside a Rails Project but to install the l;atest gem :
+
+    `gem install ar_loader`
+
+To pull the tasks in, add call in your Rakefile to  :
+
+    ArLoader::require_tasks
+
+## Example Spreadsheet
+
+  An example Spreadsheet with headers and comments, suitable for giving to Clients
+  to populate, can be found in test/examples/DemoSpreadsheet.xls
+
+## Features
+
+- *Direct Excel support*
+
+  Includes a wrapper around MS Excel via Apache POI, which
+  enables Products to be loaded directly from Excel via JRuby. No need to save to CSV first.
+
+  The java jars e.g - 'poi-3.6.jar' - are included.
+
+- *Semi-Smart Name Lookup*
+
+  Includes helper classes that find and store details of all possible associations on an AR class.
+  Given a user supplied name, attempts to find the requested association.
+
+  Example usage, load from a file or spreadsheet where the column names are only
+  an approximation of the actual associations, so given 'Product Properties' heading,
+  finds real association 'product_properties' to send or call on the AR object
+
+- *Associations*
+
+  Can handle 'many' type associations and enables multiple association objects to
+  be added via single entry (column). See Details section.
+
+- *Spree Rake Tasks*
+
+  Rake tasks provided for Spree loading - currently supports Product with associations,
+  and Image loading.
+
+  **Product loading from Excel specifically requires JRuby**. 
+
+  Example command lines:
+
+    rake excel_load input=vendor\extensions\autotelik\fixtures\ExampleInfoWeb.xls
+
+    rake excel_load input=C:\MyProducts.xls verbose=true'
+
+- *Seamless Image loading can be achieved by ensuring SKU or class Name features in Image filename.
+
+  Lookup is performed either via the SKU being prepended to the image name, or by the image name being equal to the **name attribute** of the klass in question.
+
+  Images can be attached to any class defined with a suitable association. The class to use can be configured in rake task via
+  parameter klass=Xyz.
+
+  In the Spree tasks, this defaults to Product, so attempts to attach Image to a Product via Product SKU or Name.
+ 
+  Image loading **does not** specifically require JRuby
+
+  A report is generated in the current working directory detailing any Images in the paths that could not be matched with a Product.
+
+  Example cmd lines :
+
+      rake image_load input=vendor\extensions\autotelik\lib\fixtures
+      rake image_load input="C:\images\Paintings' dummy=true
+      rake image_load input="C:\images\TaxonIcons" skip_if_no_assoc=true klass=Taxon
+
+## Example Wrapper Tasks for Spree Site Extension
+
+These tasks show how to write your own high level wrapper task, that will seed the database from multiple spreedsheets.
+
+The images in this example have been named with the SKU present in name (separated by whitespace) e.g "PRINT_001 Stonehenge.jpg"
+
+A report is generated in the current working directory detailing any Images in the paths that could not be matched with a Product.
+
+    require 'ar_loader'
+
+    namespace :mysite do
+
+    desc "Load Products for site"
+    task :load, :needs => [:environment] do |t, args|
+
+      [ "vendor/extensions/site/db/seed/Paintings.xls",
+        "vendor/extensions/site/db/seed/Drawings.xls"
+      ].each do |x|
+        Rake::Task['autotelik:excel_load'].execute(
+          :input => x,
+          :verbose => true,
+          :sku_prefix => ""
+        )
+      end
+    end
+
+    desc "Load Images for site based on SKU"
+    task :load_images, :clean, :dummy, :needs => [:environment] do |t, args|
+
+      if(args[:clean])
+        Image.delete_all
+        FileUtils.rm_rf( "public/assests/products" )
+      end
+
+      ["01_paintings_jpegs", "02_drawings_jpegs"].each do |x|
+
+        # image names start with associated Product SKU,
+        # skip rather then exit if no matching product found
+
+        Rake::Task['autotelik:image_load'].execute(
+          :input => "/my_site_load_info//#{x}",
+          :dummy => args[:dummy],
+          :verbose => false, :sku => true, :skip_if_no_assoc => true
+        )  
+      end
+    end
+
+## Details
+
+### Associations
+
+A single association column can contain multiple name/value sets in default form :
+
+    Name1:value1, value2|Name2:value1, value2, value3|Name3:value1, value2 etc
+
+So for example a Column for an 'Option Types' association on a Product,
+ could contain 2 options with a number of values each :
+
+'Option Types'
+    size:small,medium,large|colour:red,white
+    size:small|colour:blue,red,white
+
+### Properties
+
+The properties to associate with this product.
+Properties are for small snippets of text, shared across many products,
+and are for display purposes only.
+
+An optional display value can be supplied to supplement the displayed text.
+
+As for all associations can contain multiple name/value sets in default form :
+
+    Property:display_value|Property:display_value
+
+Example - No values :
+    manufacturer|standard
+
+Example - Display  values :
+    manufacturer:somebody else plc|standard:ISOBlah21
+
+## TODO
+
+  - Make more generic, so have smart switching to Ruby and directly support csv,
+    when JRuby and/or Excel not available.
+
+  - Smart sorting of column processing order ....
+
+    Does not currently ensure mandatory columns (for valid?) processed first.
+    Since Product needs saving before associations can be processed, user currently
+    needs to ensure SKU, name, price columns are among first columns
+
+## License
+
+Copyright:: (c) Autotelik Media Ltd 2011
+
+Author ::   Tom Statter
+
+Date ::     Feb 2011
+
+The MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/Rakefile

@@ -0,0 +1,76 @@
+require 'rubygems'
+require 'rake'
+require 'rake/clean'
+require 'rake/gempackagetask'
+require 'rake/rdoctask'
+require 'rake/testtask'
+require "lib/ar_loader"
+
+# Copyright:: (c) Autotelik Media Ltd 2011
+# Author ::   Tom Statter
+# Date ::     Aug 2010
+#
+# License::   MIT - Free, OpenSource
+#
+# Details::   Gem::Specification for Active Record Loader gem.
+#
+#             Specifically enabled for uploading Spree products but easily
+#             extended to any AR model.
+#
+#             Currently support direct access to Excel Spreedsheets via JRuby
+#
+#             TODO - Switch for non JRuby Rubies, enable load via CSV file instead of Excel.
+#
+ArLoader::require_tasks
+
+spec = Gem::Specification.new do |s|
+  s.name = ArLoader.gem_name
+  s.version = ArLoader.gem_version
+  s.has_rdoc = true
+  s.extra_rdoc_files = ['README.markdown', 'LICENSE']
+  s.summary = 'File based loader for Active Record models'
+  s.description = 'A file based loader for Active Record models. Seed database directly from Excel/CSV. Includes rake support for Spree'
+  s.author = 'thomas statter'
+  s.email = 'rubygems@autotelik.co.uk'
+  s.date = DateTime.now.strftime("%Y-%m-%d")
+  s.homepage = %q{http://www.autotelik.co.uk}
+
+  # s.executables = ['your_executable_here']
+  s.files = %w(LICENSE README.markdown Rakefile) + Dir.glob("{lib,spec,tasks}/**/*")
+  s.require_path = "lib"
+  s.bindir = "bin"
+end
+
+Rake::GemPackageTask.new(spec) do |p|
+  p.gem_spec = spec
+  p.need_tar = true
+  p.need_zip = true
+end
+
+Rake::RDocTask.new do |rdoc|
+  files =['README.markdown', 'LICENSE', 'lib/**/*.rb']
+  rdoc.rdoc_files.add(files)
+  rdoc.main = "README.markdown" # page to start on
+  rdoc.title = "ARLoader Docs"
+  rdoc.rdoc_dir = 'doc/rdoc' # rdoc output folder
+  rdoc.options << '--line-numbers'
+end
+
+Rake::TestTask.new do |t|
+  t.test_files = FileList['test/**/*.rb']
+end
+
+# Add in our own Tasks
+
+# Long parameter lists so ensure rake -T produces nice wide output
+ENV['RAKE_COLUMNS'] = '180'
+
+desc 'Build gem and install in one step'
+task :pik_install, :needs => [:gem]  do |t, args|
+
+  puts "Installing version #{ArLoader.gem_version}"
+
+  gem = "#{ArLoader.gem_name}-#{ArLoader.gem_version}.gem"
+  cmd = "pik gem install --no-ri --no-rdoc pkg\\#{gem}"
+  system(cmd)
+end

data/lib/VERSION

@@ -0,0 +1 @@
+0.0.4 

data/lib/ar_loader.rb

@@ -0,0 +1,53 @@
+# Copyright:: (c) Autotelik Media Ltd 2011
+# Author ::   Tom Statter
+# Date ::     Aug 2010
+# License::   TBD. Free, Open Source. MIT ?
+#
+# Details::   Active Record Loader
+#
+module ArLoader
+
+  def self.gem_version
+    @gem_version ||= File.read( File.join( root_path, 'lib', 'VERSION') ).chomp
+    @gem_version
+  end
+
+  def self.gem_name
+    "ar_loader"
+  end
+
+  def self.root_path
+    File.expand_path("#{File.dirname(__FILE__)}/..")
+  end
+
+  def self.require_libraries
+
+    loader_libs = %w{ lib  }
+
+    # Base search paths - these will be searched recursively and any xxx.rake files autoimported
+    loader_paths = []
+
+    loader_libs.each {|lib| loader_paths << File.join(root_path, lib) }
+
+    # Define require search paths, any dir in here will be added to LOAD_PATH
+
+    loader_paths.each do |base|
+      $:.unshift base  if File.directory?(base)
+      Dir[File.join(base, '**', '**')].each do |p|
+        if File.directory? p
+          $:.unshift p
+        end
+      end
+    end
+  end
+
+  def self.require_tasks
+    # Long parameter lists so ensure rake -T produces nice wide output
+    ENV['RAKE_COLUMNS'] = '180'
+    base = File.join(root_path, 'tasks', '**')
+    Dir["#{base}/*.rake"].sort.each { |ext| load ext }
+  end
+  
+end
+
+ArLoader::require_libraries

data/lib/engine/file_definitions.rb

@@ -0,0 +1,353 @@
+# Copyright:: (c) Autotelik Media Ltd 2011
+# Author ::   Tom Statter
+# Date ::     Jan 2011
+# License::   MIT
+#
+# Details::   This module acts as helpers for defining input/output file formats as classes.
+#
+# It provides a simple interface to define a file structure - field by field.
+#
+# By defining the structure, following methods and attributes are mixed in :
+#
+#   An attribute, with accessor for each field/column.
+#   Parse a line, assigning values to each attribute.
+#   Parse an instance of that file line by line, accepts a block in which data can be processed.
+#   Method to split a file by field.
+#   Method to perform replace operations on a file by field and value.
+#
+# Either delimited or a fixed width definition can be created via macro-like class methods :
+#
+#   create_field_definition [field_list]
+#
+#   create_fixed_definition {field => range }
+#
+# Member attributes, with getters and setters, can be added for each field defined above via class method :
+#
+#   create_field_attr_accessors
+#
+# USAGE :
+#
+# Create a class that contains definition of a file.
+#
+#   class ExampleFixedWith  < FileDefinitionBase
+#     create_fixed_definition(:name => (0..7), :value => (8..15), :ccy => (16..18), :dr_or_cr => (19..19) )
+#
+#     create_field_attr_accessors
+#   end
+#
+#   class ExampleCSV < FileDefinitionBase
+#     create_field_definition %w{abc def  ghi jkl}
+#
+#     create_field_attr_accessors
+#   end
+#
+# Any instance can then be used to parse the defined file type, with each field or column value
+# being assigned automatically to the associated instance variable.
+#
+#   line = '1,2,3,4'
+#   x = ExampleCSV.new( line )
+#
+#   assert x.responds_to? :jkl
+#   assert_equal x.abc, '1'
+#   assert_equal x.jkl.to_i, 4
+#
+module FileDefinitions
+
+  include Enumerable
+
+  attr_accessor :key
+  attr_accessor :current_line
+
+  # Set the delimiter to use when splitting a line - can be either a String, or a Regexp
+  attr_writer   :field_delim
+
+  def initialize( line  = nil )
+    @key   = String.new
+    parse(line) unless line.nil?
+  end
+
+  def self.included(base)
+    base.extend(ClassMethods)
+    subclasses << base
+  end
+
+  def self.subclasses
+    @subclasses ||=[]
+  end
+
+
+  # Return the field delimiter used when splitting a line
+  def field_delim
+    @field_delim || ','
+  end
+
+  # Parse each line of a file based on the field definition, yields self for each successive line
+  #
+  def each( file )
+    File::new(file).each_line do |line|
+      parse( line )
+      yield self
+    end
+  end
+
+  def fields
+    @fields = self.class.field_definition.collect {|f| instance_variable_get "@#{f}" }
+    @fields
+  end
+
+  def to_s
+    fields.join(',')
+  end
+
+  module ClassMethods
+
+    # Helper to generate methods to store and return the complete list of fields
+    # in this File definition (also creates member @field_definition) and parse a line.
+    #
+    # e.g create_field_definition %w{ trade_id  drOrCr ccy costCentre postingDate amount }
+    #
+    def create_field_definition( *fields )
+      instance_eval <<-end_eval
+          @field_definition ||= %w{ #{fields.join(' ')} }
+          def field_definition
+            @field_definition
+          end
+      end_eval
+
+      class_eval <<-end_eval
+        def parse( line )
+          @current_line = line
+          before_parse  if respond_to? :before_parse
+          @current_line.split(field_delim()).each_with_index {|x, i| instance_variable_set(\"@\#{self.class.field_definition[i]}\", x) }
+          after_parse  if respond_to? :after_parse
+          generate_key if respond_to? :generate_key
+        end
+      end_eval
+    end
+
+    def add_field(field, add_accessor = true)
+      @field_definition ||= []
+      @field_definition << field.to_s
+      attr_accessor field  if(add_accessor)
+    end
+
+
+    # Helper to generate methods that return the complete list of fixed width fields
+    # and associated ranges in this File definition, and parse a line.
+    # e.g create_field_definition %w{ trade_id  drOrCr ccy costCentre postingDate amount }
+    #
+    def create_fixed_definition( field_range_map )
+      raise ArgumentError.new('Please supply hash to create_fixed_definition') unless field_range_map.is_a? Hash
+
+      keys = field_range_map.keys.collect(&:to_s)
+      string_map = Hash[*keys.zip(field_range_map.values).flatten]
+
+      instance_eval <<-end_eval
+        def fixed_definition
+          @fixed_definition ||= #{string_map.inspect}
+          @fixed_definition
+        end
+      end_eval
+
+      instance_eval <<-end_eval
+        def field_definition
+          @field_definition ||= %w{  #{keys.join(' ')} }
+          @field_definition
+        end
+      end_eval
+
+      class_eval <<-end_eval
+        def parse( line )
+          @current_line = line
+          before_parse  if respond_to? :before_parse
+          self.class.fixed_definition.each do |key, range|
+            instance_variable_set(\"@\#{key}\", @current_line[range])
+          end
+          after_parse  if respond_to? :after_parse
+          generate_key if respond_to? :generate_key
+        end
+      end_eval
+
+    end
+
+    # Create accessors for each field
+    def create_field_attr_accessors
+      self.field_definition.each {|f| attr_accessor f}
+    end
+
+
+    ###############################
+    # PARSING + FILE MANIPULATION #
+    ###############################
+
+    # Parse a complete file and return array of self, one per line
+    def parse_file( file, options = {} )
+      limit = options[:limit]
+      count = 0
+      lines = []
+      File::new(file).each_line do |line|
+        break if limit && ((count += 1) > limit)
+        lines << self.new( line )
+      end
+      lines
+    end
+
+
+
+    # Split a file, whose field definition is represented by self,
+    # into seperate streams, based on the values of one if it's fields.
+    #
+    # Writes the results, one file per split stream, to directory specified by output_path
+    #
+    # Options:
+    #
+    #   :keys       => Also write split files of the key fields
+    #
+    #   :filter     => Optional Regular Expression to act as filter be applid to the field.
+    #                  For example split by Ccy but filter to only include certain ccys pass
+    #                  filter => '[GBP|USD]'
+    #
+    def split_on_write( file_name, field, output_path, options = {} )
+
+      path = output_path || '.'
+
+      filtered = split_on( file_name, field, options )
+
+      unless filtered.empty?
+        log :info, "Writing seperate streams to #{path}"
+
+        filtered.each { |strm, objects| RecsBase::write( {"keys_#{field}_#{strm}.csv" => objects.collect(&:key).join("\n")}, path) } if(options.key?(:keys))
+
+        filtered.each { |strm, objects| RecsBase::write( {"#{field}_#{strm}.csv" => objects.collect(&:current_line).join("\n")}, path) }
+      end
+    end
+
+    # Split a file, whose field definition is represented by self,
+    # into seperate streams, based on one if it's fields.
+    #
+    # Returns a map of Field value => File def object
+    #
+    # We return the File Def object as this is now enriched, e.g with key fields, compared to the raw file.
+    #
+    # Users can get at the raw line simply by calling the line() method on File Def object
+    #
+    # Options:
+    #
+    #   :output_path => directory to write the individual streams files to
+    #
+    #   :filter      => Optional Regular Expression to act as filter be applid to the field.
+    #                  For example split by Ccy but filter to only include certain ccys pass
+    #                  filter => 'GBP|USD|EUR'
+    #
+    def split_on( file_name, field, options = {} )
+
+      regex = options[:filter] ? Regexp.new(options[:filter]) : nil
+
+      log :debug, "Using REGEX: #{regex.inspect}" if regex
+
+      filtered = {}
+
+      if( self.new.respond_to?(field) )
+
+        log :info, "Splitting on #{field}"
+
+        File.open( file_name ) do |t|
+          t.each do |line|
+            next unless(line && line.chomp!)
+            x = self.new(line)
+
+            value = x.send( field.to_sym )    # the actual field value from the specified field column
+            next if value.nil?
+
+            if( regex.nil? || value.match(regex) )
+              filtered[value] ? filtered[value] << x : filtered[value] = [x]
+            end
+          end
+        end
+      else
+        log :warn, "Field [#{field}] nor defined for file definition #{self.class.name}"
+      end
+
+      if( options[:sort])
+        filtered.values.each( &:sort )
+        return filtered
+      end
+      return filtered
+    end
+
+    # Open and parse a file, replacing a value in the specfied field.
+    # Does not update the file itself. Does not write a new output file.
+    #
+    # Returns :
+    #   1) full collection of updated lines
+    #   2) collection of file def objects (self), with updated value.
+    #
+    # Finds values matching old_value in given map
+    #
+    # Replaces matches with new_value in map.
+    #
+    # Accepts more than one field, if files is either and array of strings
+    # or comma seperated list of fields.
+    #
+    def file_set_field_by_map( file_name, fields, value_map, regex = nil )
+
+      lines, objects = [],[]
+
+      if fields.is_a?(Array)
+        attribs = fields
+      else
+        attribs = "#{fields}".split(',')
+      end
+
+      attribs.collect! do |attrib|
+        raise BadConfigError.new("Field: #{attrib} is not a field on #{self.class.name}") unless self.new.respond_to?(attrib)
+      end
+
+      log :info, "#{self.class.name} - updating field(s) #{fields} in #{file_name}"
+
+      File.open( file_name ) do |t|
+        t.each do |line|
+          if line.chomp.empty?
+            lines << line
+            objects << self.new
+            next
+          end
+          x = self.new(line)
+
+          attribs.each do |a|
+            old_value = x.instance_variable_get( "@#{a}" )
+            x.instance_variable_set( "@#{a}", value_map[old_value] ) if value_map[old_value] || (regex && old_value.keys.detect {|k| k.match(regx) })
+          end
+
+          objects << x
+          lines << x.to_s
+        end
+      end
+
+      return lines, objects
+    end
+  end   # END class methods
+
+  # Open and parse a file, replacing a value in the specfied field.
+  # Does not update the file itself. Does not write a new output file.
+  #
+  # Returns :
+  #   1) full collection of updated lines
+  #   2) collection of file def objects (self), with updated value.
+  #
+  # Finds values matching old_value, and also accepts an optional regex for more powerful
+  # matching strategies of values on the specfified field.
+  #
+  # Replaces matches with new_value.
+  #
+  # Accepts more than one field, if files is either and array of strings
+  # or comma seperated list of fields.
+  #
+  def file_set_field( file_name, field, old_value, new_value, regex = nil )
+
+    map = {old_value => new_value}
+
+    return file_set_field_by_map(file_name, field, map, regex)
+  end
+
+end