datashift 0.13.0 → 0.14.0

data/README.markdown

@@ -1,5 +1,10 @@
 ##  DataShift 
 
+- [Features](#features)
+- [Installation](#installation)
+- [Active Record - Import/Export](#Active Record - Import/Export)
+- [License](#license)
+
 Provides tools to shift data between Excel/CSV files and Rails projects and Ruby applications
 
 Import and export models fully with all associations.
@@ -7,7 +12,8 @@ Import and export models fully with all associations.
 Comprehensive Wiki here : **https://github.com/autotelik/datashift/wiki**
 
 Specific command line tools and full Product loading for Spree E-Commerce 
-now seperate gem at [datashift_spree](https://github.com/autotelik/datashift_spree "Datashift Spree")
+now separate gem at [datashift_spree](https://github.com/autotelik/datashift_spree "Datashift Spree")
+
 
 ### Features
 
@@ -40,11 +46,15 @@ Many example Spreadsheets/CSV files in spec/fixtures, fully documented with comm
 
 Add gem 'datashift' to your Gemfile/bundle or use ```gem install```
 
-    ```ruby gem 'datashift' ```
+```ruby 
+gem 'datashift' 
+```
 
 For Spree support also add :
 
-    ```ruby gem 'datashift_spree' ```
+```ruby 
+gem 'datashift_spree'
+```
 
 To use :
 
@@ -106,6 +116,26 @@ Provides high level  tasks for exporting data to various sources, currently .xls
 
     bundle exec thor datashift:export:excel model=BlogPost result=BlogExport.xls 
 
+  
+Import based on column headings with *Semi-Smart Name Lookup*
+
+  On import, first a dictionary of all possible attributes and associations is created for the AR class.
+  
+  This enables lookup, of a user supplied name (column heading), managing white space, pluralisation etc .
+
+  Example usage, load from a file or spreadsheet where the column names are only
+  an approximation of the actual associations, so given 'Product Properties' heading,
+  finds real association 'product_properties' to send or call on the AR object
+
+
+Can import/export 'belongs_to, 'has_many' and 'has_one' associations, including assignment of multiple objects
+via either multiple columns, or via a DSL for creating multiple entries in a single (column). 
+
+The DSL can also be used to define which fields to lookup associations, and assign values to other fields.
+
+See Wiki for more details on DSL syntax.
+
+Supports inclusion of delegated attributes and normal instance methods as column headings.
 
 The library can be easily extended with Loaders to deal with non trivial cases,
  for example when multiple lookups required to find right association.
@@ -135,22 +165,8 @@ This data can be exported directly to CSV or Excel/OpenOffice spreadsheets.
   Column headings contain comments with full descriptions and instructions on syntax. 
 
 
-## Features
-
-- *Associations*
-
-  Can import/export 'belongs_to, 'has_many' and 'has_one' associations, including assignment of multiple objects
-  via either multiple columns, or via a DSL for creating multiple entries in a single (column). 
+## Excel
 
-  The DSL can also be used to define which fields to lookup associations, and assign values to other fields.
-
-  See Wiki for more details on DSL syntax.
-
-  Supports delegated attributes.
-
-- *High level wrappers around applications including Excel and Word
-
-  Quickly and easily access common enterprise applications through Ruby
 
   MS Excel itself does not need to be installed.
 
@@ -160,55 +176,10 @@ This data can be exported directly to CSV or Excel/OpenOffice spreadsheets.
   
   The required POI jars are already included.
 
-- *Direct Excel export*
-
   Excel/OpenOffice spreadsheets are heavily used in many sectors, so direct support makes it
-  easier and quicker to migrate your client's data into a Rails/ActiveRecord project.
-
-  No need to save to CSV or map to YAML.
-  
-- *Semi-Smart Name Lookup*
-
-  Includes helper classes that find and store details of all possible associations on an AR class.
-  Given a user supplied name, attempts to find the requested association.
-
-  Example usage, load from a file or spreadsheet where the column names are only
-  an approximation of the actual associations, so given 'Product Properties' heading,
-  finds real association 'product_properties' to send or call on the AR object
-
-
-
-- *Thor Tasks*
-
-  High level Thor CLIs are provided, only required to supply model class, and file location :
-
-    thor datashift:import:excel model=MusicTrack input=MyTrackListing.xls
-
-
-- *Spree Tasks*
-
-  Spree's product associations are non trivial so specific Rake tasks are also provided for loading Spree Producta
-  with all associations and Image loading.
-
-    thor datashift:spree:products input=C:\MyProducts.xls
-
-
-- *Seamless Spree Image loading can be achieved by ensuring SKU or class Name features in Image filename.
-
-  Lookup is performed either via the SKU being prepended to the image name, or by the image name being equal to the **name attribute** of the klass in question.
-
-  Images can be attached to any class defined with a suitable association. The class to use can be configured in rake task via
-  parameter klass=Xyz.
-
-  In the Spree tasks, this defaults to Product, so attempts to attach Image to a Product via Product SKU or Name.
-
-  A report is generated in the current working directory detailing any Images in the paths that could not be matched with a Product.
-
-    thor  datashift:spree:images input=C:\images\product_images skip_if_no_assoc=true
-
-    thor datashift:spree:images input=C:\images\taxon_icons skip_if_no_assoc=true klass=Taxon
+  easier and quicker to migrate your client's data into a Rails/ActiveRecord project,
+  without converting first to CSV or YAML.
 
-## Import to Active Record
 
 ### Associations
 
@@ -237,7 +208,6 @@ During loading, a call to find_all_by_reference will be made, picking up the 2 c
   - Look at implementing import/export API using something like https://github.com/ianwhite/orm_adapter 
     rather than active record, so we can support additional ORMs
 
-  - Create separate Spree extension to support import/export via the admin gui
     
 ## License
 

data/VERSION

@@ -1 +1 @@
-0.13.0
+0.14.0

data/lib/applications/jexcel_file.rb

@@ -95,10 +95,14 @@ if(DataShift::Guards::jruby?)
         
         return create_sheet_and_set_styles( sheet_name )
       else 
-        if (@workbook.getSheetIndex(sheet_name) < 0)  #Check sheet doesn't already exist
-          return create_sheet_and_set_styles( sheet_name )
+        
+        name = sanitize_sheet_name( sheet_name )
+        
+        puts "WTF #{name}"
+        if (@workbook.getSheetIndex(name) < 0)  #Check sheet doesn't already exist
+          return create_sheet_and_set_styles( name )
         else
-          activate_sheet(sheet_name)
+          activate_sheet(name)
         end
       end
     end
@@ -205,9 +209,12 @@ if(DataShift::Guards::jruby?)
     private
        
     def create_sheet_and_set_styles( sheet_name )
-      @sheet = @workbook.createSheet( sanitize_sheet_name(sheet_name) )
+      
+      name = sanitize_sheet_name(sheet_name)
+      
+      @sheet = @workbook.createSheet( name )
 
-      @patriarchs.store(sheet_name, @sheet.createDrawingPatriarch())
+      @patriarchs.store(name, @sheet.createDrawingPatriarch())
      
       @date_style = @workbook.createCellStyle
       @date_style.setDataFormat( JExcelFile::date_format )

data/lib/datashift.rb

@@ -32,21 +32,9 @@
 #
 #     DataShift::load_commands
 #
-require 'rbconfig'
-require 'guards'
 
 module DataShift
 
-  if(Guards::jruby?)
-    require 'java'
-      
-    class Object
-      def add_to_classpath(path)
-        $CLASSPATH << File.join( DataShift.root_path, 'lib', path.gsub("\\", "/") )
-      end
-    end
-  end
-
   def self.gem_version
     unless(@gem_version)
       if(File.exists?('VERSION'))
@@ -126,4 +114,21 @@ module DataShift
   
 end
 
-DataShift::require_libraries
+DataShift::require_libraries
+
+require 'datashift/guards'
+require 'datashift/method_detail'
+require 'datashift/method_dictionary'
+require 'datashift/method_mapper'
+
+module DataShift
+  if(Guards::jruby?)
+    require 'java'
+      
+    class Object
+      def add_to_classpath(path)
+        $CLASSPATH << File.join( DataShift.root_path, 'lib', path.gsub("\\", "/") )
+      end
+    end
+  end
+end

data/lib/datashift/delimiters.rb

@@ -17,7 +17,6 @@ module DataShift
     # Support multiple associations being added to a base object to be specified in a single column.
     # 
     # Entry represents the association to find via supplied name, value to use in the lookup.
-    # Can contain multiple lookup name/value pairs, separated by multi_assoc_delim ( | )
     # 
     # Default syntax :
     #

data/lib/datashift/method_detail.rb

@@ -20,10 +20,7 @@ module DataShift
   class MethodDetail
 
     include DataShift::Logging
-    
-    include DataShift::Populator
-    extend DataShift::Populator
-    
+      
     def self.supported_types_enum
       @type_enum ||= Set[:assignment, :belongs_to, :has_one, :has_many]
       @type_enum
@@ -33,10 +30,7 @@ module DataShift
       @assoc_type_enum ||= Set[:belongs_to, :has_one, :has_many]
       @assoc_type_enum
     end
-    
-    # When looking up an association, try each of these in turn till a match
-    #  i.e find_by_name .. find_by_title and so on, lastly try the raw id
-    @@insistent_find_by_list ||= [:name, :title, :id]
+
 
     # Name is the raw, client supplied name
     attr_accessor :name
@@ -115,60 +109,6 @@ module DataShift
       @operator_class
     end
 
-    def assign(record, value )
-      
-      @current_value = value
-
-      # logger.info("WARNING nil value supplied for Column [#{@name}]") if(@current_value.nil?)
-    
-      if( operator_for(:belongs_to) )
-      
-        #puts "DEBUG : BELONGS_TO : #{@name} : #{operator} - Lookup #{@current_value} in DB"
-        insistent_belongs_to(record, @current_value)
-
-      elsif( operator_for(:has_many) )
-        
-        #puts "DEBUG : VALUE TYPE [#{value.class.name.include?(operator.classify)}] [#{ModelMapper.class_from_string(value.class.name)}]" unless(value.is_a?(Array))
-     
-        # The include? check is best I can come up with right now .. to handle module/namespaces
-        # TODO - can we determine the real class type of an association
-        # e.g given a association taxons, which operator.classify gives us Taxon, but actually it's Spree::Taxon
-        # so how do we get from 'taxons' to Spree::Taxons ? .. check if further info in reflect_on_all_associations
-
-        if(value.is_a?(Array) || value.class.name.include?(operator.classify))
-          record.send(operator) << value
-        else
-          puts "ERROR #{value.class} - Not expected type for has_many #{operator} - cannot assign"
-        end
-
-      elsif( operator_for(:has_one) )
-
-        #puts "DEBUG : HAS_MANY :  #{@name} : #{operator}(#{operator_class}) - Lookup #{@current_value} in DB"
-        if(value.is_a?(operator_class))
-          record.send(operator + '=', value)
-        else
-          logger.error("ERROR #{value.class} - Not expected type for has_one #{operator} - cannot assign")
-          # TODO -  Not expected type - maybe try to look it up somehow ?"
-          #insistent_has_many(record, @current_value)
-        end
-
-      elsif( operator_for(:assignment) && @col_type )
-        #puts "DEBUG : COl TYPE defined for #{@name} : #{@assignment} => #{@current_value} #{@col_type.type}"
-       # puts "DEBUG : Column [#{@name}] : COl TYPE CAST: #{@current_value} => #{@col_type.type_cast( @current_value ).inspect}"
-        record.send( operator + '=' , @col_type.type_cast( @current_value ) )
-
-        #puts "DEBUG : MethodDetails Assignment RESULT: #{record.send(operator)}"
-
-      elsif( operator_for(:assignment) )
-        #puts "DEBUG : Column [#{@name}] : Brute force assignment of value  #{@current_value}"
-        # brute force case for assignments without a column type (which enables us to do correct type_cast)
-        # so in this case, attempt straightforward assignment then if that fails, basic ops such as to_s, to_i, to_f etc
-        insistent_assignment(record, @current_value)
-      else
-        puts "WARNING: No operator found for assignment on #{self.inspect} for Column [#{@name}]"
-      end
-    end
-
     def pp
       "#{@name} => #{operator}"
     end
@@ -222,12 +162,9 @@ module DataShift
         end
       end
     end
-
-    def insistent_assignment( record, value )
-      Populator::insistent_assignment( record, value, operator)
-    end
     
-    private
+  private
+  
     # Return the operator's expected class, if can be derived, else nil
     def get_operator_class()
       if(operator_for(:has_many) || operator_for(:belongs_to) || operator_for(:has_one))  

data/lib/datashift/method_details_manager.rb

@@ -23,10 +23,15 @@ module DataShift
     end
     
     def add(method_details)
+      #puts "DEBUG: MGR - Add {#method_details.operator_type}\n#{method_details.inspect}"
       @method_details[method_details.operator_type.to_sym] ||= {}
+      
+      # Mapped by Type and MethodDetail name
+      @method_details[method_details.operator_type.to_sym][method_details.name] = method_details
+      
+      # Helper list of all available by type
       @method_details_list[method_details.operator_type.to_sym] ||= []
        
-      @method_details[method_details.operator_type.to_sym][method_details.name] = method_details
       @method_details_list[method_details.operator_type.to_sym] << method_details
       @method_details_list[method_details.operator_type.to_sym].uniq!
     end
@@ -38,10 +43,11 @@ module DataShift
     def find(name, type)
       method_details = get(type)
      
-      method_details ?  method_details[name] : nil
+      method_details ? method_details[name] : nil
     end
     
-    # type is expected to be one of MethodDetail::supportedtype_enum
+    # type is expected to be one of MethodDetail::supported_types_enum
+    # Returns all MethodDetail(s) for supplied type
     def get( type )
       @method_details[type.to_sym]
     end
@@ -51,9 +57,15 @@ module DataShift
     end
     
     alias_method(:get_list_of_method_details, :get_list) 
-    
-    def get_operators( op_type )
-      get_list(op_type).collect { |md| md.operator }
+
+    # Get list of the inbound or externally supplied names
+    def get_names(type)
+      get_list(type).collect { |md| md.name }
+    end
+
+    # Get list of Rails model operators   
+    def get_operators(type)
+      get_list(type).collect { |md| md.operator }
     end
     
     alias_method(:get_list_of_operators, :get_list) 

data/lib/datashift/method_dictionary.rb

@@ -5,8 +5,6 @@
 #
 # Details::   A cache type class that stores details of all possible associations on AR classes.
 #             
-require 'method_detail'
-
 module DataShift
 
   class MethodDictionary
@@ -23,7 +21,7 @@ module DataShift
     # grouped by type of association (includes belongs_to and has_many which provides both << and = )
     # Options:
     #   :reload => clear caches and re-perform  lookup
-    #   :instance_methods => if true include instance method type assignment operators as well as model's pure columns
+    #   :instance_methods => if true include instance method type 'setters' as well as model's pure columns
     #
     def self.find_operators(klass, options = {} )
       
@@ -34,50 +32,39 @@ module DataShift
         has_many[klass] = klass.reflect_on_all_associations(:has_many).map { |i| i.name.to_s }
         klass.reflect_on_all_associations(:has_and_belongs_to_many).inject(has_many[klass]) { |x,i| x << i.name.to_s }
       end
-      
-      # puts "DEBUG: Has Many Associations:", has_many[klass].inspect
 
       # Find the belongs_to associations which can be populated via  Model.belongs_to_name = OtherArModelObject
       if( options[:reload] || belongs_to[klass].nil? )
         belongs_to[klass] = klass.reflect_on_all_associations(:belongs_to).map { |i| i.name.to_s }
       end
 
-      #puts "Belongs To Associations:", belongs_to[klass].inspect
-
       # Find the has_one associations which can be populated via  Model.has_one_name = OtherArModelObject
       if( options[:reload] || has_one[klass].nil? )
         has_one[klass] = klass.reflect_on_all_associations(:has_one).map { |i| i.name.to_s }
       end
 
-      #puts "has_one Associations:", self.has_one[klass].inspect
-
       # Find the model's column associations which can be populated via xxxxxx= value
       # Note, not all reflections return method names in same style so we convert all to
       # the raw form i.e without the '='  for consistency 
       if( options[:reload] || assignments[klass].nil? )
  
-        # TODO investigate difference with attribute_names - maybe column names can be assigned to an attribute
-        # so in terms of method calls on klass attribute_names might be safer
         assignments[klass] = klass.column_names  
+         
+        # get into consistent format with other assignments names i.e remove the = for now
+        assignments[klass] += setters(klass).map{|i| i.gsub(/=/, '')} if(options[:instance_methods])
            
-        if(options[:instance_methods] == true)
-          setters = klass.instance_methods.grep(/\w+=/).collect {|x| x.to_s }
-
-          # TODO - Since 3.2 this seems to return lots more stuff including validations which might not be appropriate
-          if(klass.respond_to? :defined_activerecord_methods)
-            setters = setters - klass.defined_activerecord_methods.to_a
-          end
-
-          # get into same format as other names 
-          assignments[klass] += setters.map{|i| i.gsub(/=/, '')}
-        end
-        
-        assignments[klass] -= has_many[klass] if(has_many[klass])
+        # Now remove all the associations
+        assignments[klass] -= has_many[klass]   if(has_many[klass])
         assignments[klass] -= belongs_to[klass] if(belongs_to[klass])
-        assignments[klass] -= self.has_one[klass] if(self.has_one[klass])
- 
+        assignments[klass] -= has_one[klass]    if(has_one[klass])
+         
+        # TODO remove assignments with id
+        # assignments => tax_id  but already in belongs_to => tax
+        
         assignments[klass].uniq!
 
+        #puts "\nDEBUG: DICT Setters\n#{assignments[klass]}\n"
+        
         assignments[klass].each do |assign|
           column_types[klass] ||= {}
           column_def = klass.columns.find{ |col| col.name == assign }
@@ -86,6 +73,18 @@ module DataShift
       end
     end
     
+    def self.setters( klass )
+          
+      # N.B In 1.8 these return strings, in 1.9 symbols.
+      # map everything to strings a
+      #setters = klass.accessible_attributes.sort.collect( &:to_s )
+      
+      # remove methodsa that start with '_'
+      @keep_only_pure_setters ||= Regexp.new(/^[a-zA-Z]\w+=/)
+      
+      setters = klass.instance_methods.grep(@keep_only_pure_setters).sort.collect( &:to_s )
+      setters.uniq
+    end
 
     def self.add( klass, operator, type = :assignment)
       method_details_mgr = get_method_details_mgr( klass )
@@ -119,14 +118,14 @@ module DataShift
       method_details_mgrs[klass] = method_details_mgr
       
     end
-   
+    
     # TODO - check out regexp to do this work better plus Inflections ??
     # Want to be able to handle any of ["Count On hand", 'count_on_hand', "Count OnHand", "COUNT ONHand" etc]
     def self.substitutions(external_name)
       name = external_name.to_s
       
       [
-        name,
+        name.downcase,
         name.tableize,
         name.gsub(' ', '_'),
         name.gsub(' ', '_').downcase,
@@ -137,18 +136,24 @@ module DataShift
       ]
     end
     
+ 
     # Find the proper format of name, appropriate call + column type for a given name.
     # e.g Given users entry in spread sheet check for pluralization, missing underscores etc
     #
     # If not nil, returned method can be used directly in for example klass.new.send( call, .... )
     #
-    def self.find_method_detail( klass, external_name )
+    def self.find_method_detail( klass, external_name, conditions = nil )
 
       method_details_mgr = get_method_details_mgr( klass )
-         
-      # md_mgr.all_available_operators.each { |l| puts "DEBUG: Mapped Method : #{l.inspect}" }      
-      substitutions(external_name).each do |n|
-      
+   
+      # first try for an exact match across all association types
+      MethodDetail::supported_types_enum.each do |t|
+        method_detail = method_details_mgr.find(external_name, t)
+        return method_detail.clone if(method_detail)
+      end  
+              
+      # Now try various alternatives of the name
+      substitutions(external_name).each do |n|    
         # Try each association type, returning first that contains matching operator with name n    
         MethodDetail::supported_types_enum.each do |t|
           method_detail = method_details_mgr.find(n, t)
@@ -165,14 +170,25 @@ module DataShift
 
       method_details_mgr = get_method_details_mgr( klass )
       
-      substitutions(external_name).each do |n|
-        method_detail = method_details_mgr.find(n, :assignment)
-        return method_detail if(method_detail && method_detail.col_type) 
+      # first try for an exact match across all association types
+      MethodDetail::supported_types_enum.each do |t|
+        method_detail = method_details_mgr.find(external_name, t)
+        return method_detail.clone if(method_detail && method_detail.col_type) 
+      end  
+              
+      # Now try various alternatives
+      substitutions(external_name).each do |n|    
+        # Try each association type, returning first that contains matching operator with name n    
+        MethodDetail::supported_types_enum.each do |t|
+          method_detail = method_details_mgr.find(n, t)
+          return method_detail.clone if(method_detail && method_detail.col_type) 
+        end  
       end
-      
+
       nil
     end
     
+    
     def self.clear
       belongs_to.clear
       has_many.clear
@@ -190,7 +206,8 @@ module DataShift
       method_details_mgrs[klass] || MethodDetailsManager.new( klass )
     end
     
-        
+    
+    # Store a Mgr per mapped klass
     def self.method_details_mgrs
       @method_details_mgrs ||= {}
       @method_details_mgrs