datashift 0.0.2 → 0.1.0

data/README.markdown

@@ -54,7 +54,11 @@ Provides high level rake tasks for importing data via ActiveRecord models into a
  from various sources, currently csv or .xls files (Excel/Open Office)
 
 
-    bundle exec rake datashift:import:excel model=BlogPost input=BlogPostImport.xls verbose=true 
+    jruby -S rake datashift:import:excel model=BlogPost input=BlogPostImport.xls verbose=true 
+
+Provides high level rake tasks for exporting data to various sources, currently .xls files (Excel/Open Office)
+
+    jruby -S rake datashift:export:excel model=BlogPost result=BlogExport.xls 
 
 
 The library can be easily extended with Loaders to deal with non trivial cases required to exchange data between
@@ -63,19 +67,19 @@ The library can be easily extended with Loaders to deal with non trivial cases r
 Spree loaders are an example, these illustrate over riding processing for specific columns with
 complicated lookup requirements.
 
-A core feature of DataShift is the MethodMapper, which provides features for collecting
+A core feature of DataShift is the MethodDictionary and MethodMapper, which provides features for collecting
 reflection information from ActiveRecord models (all different associations, including join tables with many-to-many relationships).
 
 A full picture of all possible operations on a class can be created very easily, for example ona  Blog model :
 
-    MethodMapper.find_operators( Blog )
+    MethodDictionary.find_operators( Blog )
 
-This then allows Import/Export to be achieved, by mapping the file's header and column data to MethodMapper's operators
-(i.e. the model's attributes and associations).
+This then allows Import/Export to be achieved, by mapping the file's header and column data to the found operators
+i.e. the methods to set data on model's attributes and associations.
 
 Here we retrieve the method details for a column name from a file, "Blog Date"
 
-    MethodMapper.find_method_detail( Blog, "Blog Date" )
+    MethodDictionary.find_method_detail( Blog, "Blog Date" )
 
 Loaders can use this method lookup functionality, to find the correct association for a column heading,
 and populate AR object with row data.
@@ -88,7 +92,7 @@ and whenever possible, under the bonnet the data will be cast to correct DB type
 
 Here we show how a column name from a file, "Blog Date", can be mapped to Assign a stringified date, to the blog_date column, on a new Blog object :
 
-    MethodMapper.find_method_detail( Blog, "Blog Date" ).assign( Blog.new, "Sat Jul 23 2011" )
+    MethodDictionary.find_method_detail( Blog, "Blog Date" ).assign( Blog.new, "Sat Jul 23 2011" )
 
 Because it's based on reflection against the model, can build complex relationships between tables during import/export, 
 and extending data files with new columns need not require any additional Ruby coding.
@@ -173,15 +177,12 @@ This data can be exported directly to CSV or Excel/OpenOffice spreadsheets.
 
 - *Spree Rake Tasks*
 
-  Specific Rake tasks are also provided for Spree loading - currently supports Product with associations,
-  and Image loading.
+  Spree's product associations are non trivial so specific Rake tasks are also provided for loading Spree Producta
+  with all associations and Image loading.
 
     jruby -S rake datashift:spree:products input=C:\MyProducts.xls
 
 
-  **Product loading from Excel files specifically requires JRuby (But not Excel or OLE)**.
-
-
 - *Seamless Spree Image loading can be achieved by ensuring SKU or class Name features in Image filename.
 
   Lookup is performed either via the SKU being prepended to the image name, or by the image name being equal to the **name attribute** of the klass in question.

data/Rakefile

@@ -22,14 +22,14 @@
 ## encoding: utf-8
 
 require 'rubygems'
-require 'bundler'
-begin
-  Bundler.setup(:default, :development)
-rescue Bundler::BundlerError => e
-  $stderr.puts e.message
-  $stderr.puts "Run `bundle install` to install missing gems"
-  exit e.status_code
-end
+#require 'bundler'
+#begin
+#  Bundler.setup(:default, :development)
+#rescue Bundler::BundlerError => e
+#  $stderr.puts e.message
+#  $stderr.puts "Run `bundle install` to install missing gems"
+#  exit e.status_code
+#end
 require 'rake'
 
 require 'lib/datashift'

data/VERSION

@@ -1,5 +1 @@
-0.0.2
-
-module DataShift
-   VERSION=0.0.2
-end
+0.1.0

data/datashift.gemspec

@@ -5,11 +5,11 @@
 
 Gem::Specification.new do |s|
   s.name = "datashift"
-  s.version = "0.0.2"
+  s.version = "0.1.0"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["Thomas Statter"]
-  s.date = "2012-02-12"
+  s.date = "2012-02-27"
   s.description = "A suite of tools to move data between ActiveRecord models,databases,applications like Excel/Open Office, files and projects including Spree"
   s.email = "rubygems@autotelik.co.uk"
   s.extra_rdoc_files = [
@@ -19,13 +19,12 @@ Gem::Specification.new do |s|
   ]
   s.files = [
     ".document",
-    "Gemfile",
     "LICENSE.txt",
     "README.markdown",
     "README.rdoc",
     "Rakefile",
     "VERSION",
-    "datashift-0.0.2.gem",
+    "datashift-0.1.0.gem",
     "datashift.gemspec",
     "lib/applications/jruby/jexcel_file.rb",
     "lib/applications/jruby/word.rb",
@@ -34,6 +33,7 @@ Gem::Specification.new do |s|
     "lib/datashift/file_definitions.rb",
     "lib/datashift/mapping_file_definitions.rb",
     "lib/datashift/method_detail.rb",
+    "lib/datashift/method_dictionary.rb",
     "lib/datashift/method_mapper.rb",
     "lib/generators/csv_generator.rb",
     "lib/generators/excel_generator.rb",
@@ -64,17 +64,22 @@ Gem::Specification.new do |s|
     "spec/csv_loader_spec.rb",
     "spec/datashift_spec.rb",
     "spec/db/migrate/20110803201325_create_test_bed.rb",
+    "spec/excel_exporter_spec.rb",
     "spec/excel_generator_spec.rb",
     "spec/excel_loader_spec.rb",
     "spec/file_definitions.rb",
-    "spec/fixtures/.~lock.ProjectsSingleCategories.xls#",
     "spec/fixtures/BadAssociationName.xls",
     "spec/fixtures/DemoNegativeTesting.xls",
     "spec/fixtures/ProjectsDefaults.yml",
     "spec/fixtures/ProjectsMultiCategories.xls",
+    "spec/fixtures/ProjectsMultiCategoriesHeaderLookup.xls",
     "spec/fixtures/ProjectsSingleCategories.xls",
     "spec/fixtures/SimpleProjects.xls",
     "spec/fixtures/config/database.yml",
+    "spec/fixtures/images/DEMO_001_ror_bag.jpeg",
+    "spec/fixtures/images/DEMO_002_Powerstation.jpg",
+    "spec/fixtures/images/DEMO_003_ror_mug.jpeg",
+    "spec/fixtures/images/DEMO_004_ror_ringer.jpeg",
     "spec/fixtures/interact_models_db.sqlite",
     "spec/fixtures/interact_spree_db.sqlite",
     "spec/fixtures/negative/SpreeProdMiss1Mandatory.csv",
@@ -89,10 +94,12 @@ Gem::Specification.new do |s|
     "spec/fixtures/spree/SpreeProductsMultiColumn.xls",
     "spec/fixtures/spree/SpreeProductsSimple.csv",
     "spec/fixtures/spree/SpreeProductsSimple.xls",
+    "spec/fixtures/spree/SpreeProductsWithImages.xls",
     "spec/fixtures/spree/SpreeZoneExample.csv",
     "spec/fixtures/spree/SpreeZoneExample.xls",
     "spec/fixtures/test_model_defs.rb",
     "spec/loader_spec.rb",
+    "spec/method_dictionary_spec.rb",
     "spec/method_mapper_spec.rb",
     "spec/spec_helper.rb",
     "spec/spree_generator_spec.rb",
@@ -121,42 +128,9 @@ Gem::Specification.new do |s|
     s.specification_version = 3
 
     if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
-      s.add_development_dependency(%q<rails>, ["= 3.1.3"])
-      s.add_development_dependency(%q<activerecord>, ["= 3.1.3"])
-      s.add_development_dependency(%q<activesupport>, ["= 3.1.3"])
-      s.add_development_dependency(%q<jruby-openssl>, [">= 0"])
-      s.add_development_dependency(%q<activerecord-jdbcsqlite3-adapter>, [">= 0"])
-      s.add_development_dependency(%q<spree>, ["= 0.70.3"])
-      s.add_development_dependency(%q<rspec>, [">= 0"])
-      s.add_development_dependency(%q<shoulda>, [">= 0"])
-      s.add_development_dependency(%q<rdoc>, ["~> 3.12"])
-      s.add_development_dependency(%q<bundler>, ["~> 1.0.0"])
-      s.add_development_dependency(%q<jeweler>, ["~> 1.8.3"])
     else
-      s.add_dependency(%q<rails>, ["= 3.1.3"])
-      s.add_dependency(%q<activerecord>, ["= 3.1.3"])
-      s.add_dependency(%q<activesupport>, ["= 3.1.3"])
-      s.add_dependency(%q<jruby-openssl>, [">= 0"])
-      s.add_dependency(%q<activerecord-jdbcsqlite3-adapter>, [">= 0"])
-      s.add_dependency(%q<spree>, ["= 0.70.3"])
-      s.add_dependency(%q<rspec>, [">= 0"])
-      s.add_dependency(%q<shoulda>, [">= 0"])
-      s.add_dependency(%q<rdoc>, ["~> 3.12"])
-      s.add_dependency(%q<bundler>, ["~> 1.0.0"])
-      s.add_dependency(%q<jeweler>, ["~> 1.8.3"])
     end
   else
-    s.add_dependency(%q<rails>, ["= 3.1.3"])
-    s.add_dependency(%q<activerecord>, ["= 3.1.3"])
-    s.add_dependency(%q<activesupport>, ["= 3.1.3"])
-    s.add_dependency(%q<jruby-openssl>, [">= 0"])
-    s.add_dependency(%q<activerecord-jdbcsqlite3-adapter>, [">= 0"])
-    s.add_dependency(%q<spree>, ["= 0.70.3"])
-    s.add_dependency(%q<rspec>, [">= 0"])
-    s.add_dependency(%q<shoulda>, [">= 0"])
-    s.add_dependency(%q<rdoc>, ["~> 3.12"])
-    s.add_dependency(%q<bundler>, ["~> 1.0.0"])
-    s.add_dependency(%q<jeweler>, ["~> 1.8.3"])
   end
 end
 

data/lib/applications/jruby/jexcel_file.rb

@@ -212,25 +212,37 @@ if(DataShift::Guards::jruby?)
     end
     
     # Save data from an AR record to the current row, based on the record's columns [c1,c2,c3]
-     
+    # Returns the number of the final column written to  
     def ar_to_xls_row(start_column, record)
       return unless( record.is_a?(ActiveRecord::Base))
         
+      column = start_column
       record.class.columns.each do |connection_column|    
-        ar_to_xls_cell(start_column, record, connection_column)
-        start_column += 1
+        ar_to_xls_cell(column, record, connection_column)
+        column += 1
       end
+      column
     end
     
     def ar_to_xls_cell(column, record, connection_column)  
-      datum = record.send(connection_column.name)
-
-      if(connection_column.sql_type =~ /date/) then 
-        @row.createCell(column - 1, HSSFCell::CELL_TYPE_STRING).setCellValue(datum.to_s)      
-      elsif connection_column.sql_type =~ /int/ then   
-        @row.createCell(column - 1, HSSFCell::CELL_TYPE_NUMERIC).setCellValue(datum.to_i)
-      else
-        @row.createCell(column - 1, HSSFCell::CELL_TYPE_STRING).setCellValue( datum.to_s ) 
+      begin
+        datum = record.send(connection_column.name)
+
+        if(connection_column.sql_type =~ /date/) 
+          @row.createCell(column - 1, HSSFCell::CELL_TYPE_STRING).setCellValue(datum.to_s) 
+          
+        elsif(connection_column.type == :boolean || connection_column.sql_type =~ /tinyint/) 
+          @row.createCell(column - 1, HSSFCell::CELL_TYPE_BOOLEAN).setCellValue(datum) 
+          
+        elsif(connection_column.sql_type =~ /int/) 
+          @row.createCell(column - 1, HSSFCell::CELL_TYPE_NUMERIC).setCellValue(datum.to_i)
+        else
+          @row.createCell(column - 1, HSSFCell::CELL_TYPE_STRING).setCellValue( datum.to_s ) 
+        end
+        
+      rescue => e
+        puts "Failed to export #{datum} from #{connection_column.inspect} to column #{column}"
+        puts e
       end
     end
       

data/lib/datashift/method_detail.rb

@@ -14,6 +14,38 @@ require 'to_b'
 
 module DataShift
 
+  # Stores MethodDetails for a class mapped by type
+  class MethodDetailsManager
+
+    attr_reader :method_details
+    
+    def initialize( klass )
+      @parent_class = klass
+      @method_details = {}
+    end
+    
+    def add(method_details)
+      @method_details[method_details.operator_type] ||= {}
+      @method_details[method_details.operator_type][method_details.name] = method_details
+    end
+
+    def <<(method_details)
+      add(method_details)      
+    end
+
+    def find(name, type)
+      method_details = get(type)
+     
+      method_details ?  method_details[name] : nil
+    end
+    
+    # type is expected to be one of MethodDetail::type_enum
+    def get( type )
+      @method_details[type]
+    end
+      
+  end
+  
   class MethodDetail
 
     def self.type_enum
@@ -30,14 +62,21 @@ module DataShift
 
     attr_reader :operator, :operator_type
 
-    # Store the raw (client supplied) name against the active record  klass(model), operator and types
+    # TODO make it a list/primary keys
+    attr_accessor :find_by_operator
+    
+    # Store the raw (client supplied) name against the active record  klass(model).
+    # Operator is the associated method call on klass,
+    # so client name maybe Price but true operator is price
+    # 
     # col_types can typically be derived from klass.columns - set of ActiveRecord::ConnectionAdapters::Column
 
-    def initialize(client_name, klass, operator, type, col_types = {} )
+    def initialize(client_name, klass, operator, type, col_types = {}, find_by_operator = nil )
       @klass, @name = klass, client_name
+      @find_by_operator = find_by_operator
 
       if( MethodDetail::type_enum.member?(type.to_sym) )
-        @operator_type = type
+        @operator_type = type.to_sym
       else
         raise "Bad operator Type #{type} passed to Method Detail"
       end
@@ -57,7 +96,7 @@ module DataShift
     # Return the actual operator's name for supplied method type
     # where type one of :assignment, :has_one, :belongs_to, :has_many etc
     def operator_for( type )
-      return operator if(@operator_type == type)
+      return operator if(@operator_type == type.to_sym)
       nil
     end
 
@@ -105,7 +144,7 @@ module DataShift
       
       @current_value = value
 
-     # logger.info("WARNING nil value supplied for Column [#{@name}]") if(@current_value.nil?)
+      # logger.info("WARNING nil value supplied for Column [#{@name}]") if(@current_value.nil?)
     
       if( operator_for(:belongs_to) )
       

data/lib/datashift/method_dictionary.rb

@@ -0,0 +1,210 @@
+# Copyright:: (c) Autotelik Media Ltd 2012
+# Author ::   Tom Statter
+# Date ::     Aug 2010
+# License::   MIT
+#
+# Details::   A cache type class that stores details of all possible associations on AR classes.
+#             
+require 'method_detail'
+
+module DataShift
+
+  class MethodDictionary
+
+    def initialize
+    end
+
+
+    # Create simple picture of all the operator names for assignment available on an AR model,
+    # grouped by type of association (includes belongs_to and has_many which provides both << and = )
+    # Options:
+    #   :reload => clear caches and re-perform  lookup
+    #   :instance_methods => if true include instance method type assignment operators as well as model's pure columns
+    #
+    def self.find_operators(klass, options = {} )
+
+      # Find the has_many associations which can be populated via <<
+      if( options[:reload] || has_many[klass].nil? )
+        has_many[klass] = klass.reflect_on_all_associations(:has_many).map { |i| i.name.to_s }
+        klass.reflect_on_all_associations(:has_and_belongs_to_many).inject(has_many[klass]) { |x,i| x << i.name.to_s }
+      end
+      # puts "DEBUG: Has Many Associations:", has_many[klass].inspect
+
+      # Find the belongs_to associations which can be populated via  Model.belongs_to_name = OtherArModelObject
+      if( options[:reload] || belongs_to[klass].nil? )
+        belongs_to[klass] = klass.reflect_on_all_associations(:belongs_to).map { |i| i.name.to_s }
+      end
+
+      #puts "Belongs To Associations:", belongs_to[klass].inspect
+
+      # Find the has_one associations which can be populated via  Model.has_one_name = OtherArModelObject
+      if( options[:reload] || self.has_one[klass].nil? )
+        has_one[klass] = klass.reflect_on_all_associations(:has_one).map { |i| i.name.to_s }
+      end
+
+      #puts "has_one Associations:", self.has_one[klass].inspect
+
+      # Find the model's column associations which can be populated via xxxxxx= value
+      # Note, not all reflections return method names in same style so we convert all to
+      # the raw form i.e without the '='  for consistency 
+      if( options[:reload] || assignments[klass].nil? )
+
+        assignments[klass] = klass.column_names
+           
+        if(options[:instance_methods] == true)
+          setters = klass.instance_methods.grep(/\w+=/).collect {|x| x.to_s }
+
+          if(klass.respond_to? :defined_activerecord_methods)
+            setters = setters - klass.defined_activerecord_methods.to_a
+          end
+
+          # get into same format as other names 
+          assignments[klass] += setters.map{|i| i.gsub(/=/, '')}
+        end
+        
+        assignments[klass] -= has_many[klass] if(has_many[klass])
+        assignments[klass] -= belongs_to[klass] if(belongs_to[klass])
+        assignments[klass] -= self.has_one[klass] if(self.has_one[klass])
+ 
+        assignments[klass].uniq!
+
+        assignments[klass].each do |assign|
+          column_types[klass] ||= {}
+          column_def = klass.columns.find{ |col| col.name == assign }
+          column_types[klass].merge!( assign => column_def) if column_def
+        end
+      end
+    end
+    
+    
+    # Build a thorough and usable picture of the operators by building dictionary of our MethodDetail
+    # objects which can be used to import/export data to objects of type 'klass'
+    #
+    def self.build_method_details( klass )
+      method_details_mgr = MethodDetailsManager.new( klass )
+         
+      assignments_for(klass).each do |n|
+        method_details_mgr << MethodDetail.new(n, klass, n, :assignment, column_types[klass])
+      end
+        
+      has_one_for(klass).each do |n|
+        method_details_mgr << MethodDetail.new(n, klass, n, :has_one)
+      end
+        
+      has_many_for(klass).each do |n|
+        method_details_mgr << MethodDetail.new(n, klass, n, :has_many)
+      end
+        
+      belongs_to_for(klass).each do |n|
+        method_details_mgr << MethodDetail.new(n, klass, n, :belongs_to)
+      end
+      
+      method_details_mgrs[klass] = method_details_mgr
+      
+    end
+        
+    # Find the proper format of name, appropriate call + column type for a given name.
+    # e.g Given users entry in spread sheet check for pluralization, missing underscores etc
+    #
+    # If not nil, returned method can be used directly in for example klass.new.send( call, .... )
+    #
+    def self.find_method_detail( klass, external_name )
+      operator = nil
+
+      # TODO - should we raise error to warn find_operators never called ?
+      md_mgr = method_details_mgrs[klass] || MethodDetailsManager.new( klass )
+         
+      name = external_name.to_s
+
+      # TODO - check out regexp to do this work better plus Inflections ??
+      # Want to be able to handle any of ["Count On hand", 'count_on_hand', "Count OnHand", "COUNT ONHand" etc]
+      [
+        name,
+        name.tableize,
+        name.gsub(' ', '_'),
+        name.gsub(' ', '_').downcase,
+        name.gsub(/(\s+)/, '_').downcase,
+        name.gsub(' ', ''),
+        name.gsub(' ', '').downcase,
+        name.gsub(' ', '_').underscore].each do |n|
+      
+        # Try each association type, returning first that contains matching operator with name n
+      
+        MethodDetail::type_enum.each do |t|
+          method_detail = md_mgr.find(n, t)
+          return method_detail if(method_detail)
+        end
+        
+      end
+
+      nil
+    end
+
+    def self.clear
+      belongs_to.clear
+      has_many.clear
+      assignments.clear
+      column_types.clear
+      has_one.clear
+      method_details_mgrs.clear
+    end
+
+    def self.column_key(klass, column)
+      "#{klass.name}:#{column}"
+    end
+    
+    def self.method_details_mgrs
+      @method_details_mgrs ||= {}
+      @method_details_mgrs
+    end
+
+    # TODO - remove use of class variables - not good Ruby design
+    def self.belongs_to
+      @belongs_to ||={}
+      @belongs_to
+    end
+
+    def self.has_many
+      @has_many ||= {}
+      @has_many
+    end
+
+    def self.has_one
+      @has_one ||= {}
+      @has_one
+    end
+
+    def self.assignments
+      @assignments ||= {}
+      @assignments
+    end
+    
+    def self.column_types
+      @column_types ||= {}
+      @column_types  
+    end
+
+
+    def self.belongs_to_for(klass)
+      belongs_to[klass] || []
+    end
+    
+    def self.has_many_for(klass)
+      has_many[klass] || []
+    end
+
+    def self.has_one_for(klass)
+      has_one[klass] || []
+    end
+
+    def self.assignments_for(klass)
+      assignments[klass] || []
+    end
+    
+    def self.column_type_for(klass, column)
+      column_types[klass] ?  column_types[klass][column] : []
+    end
+  
+  end
+
+end