datashift 0.7.0 → 0.8.0

data/README.markdown

@@ -9,9 +9,11 @@ Wiki taking shape with more info here : **https://github.com/autotelik/datashift
 ### Features
 
 Import and Export ActiveRecord models through .xls or CSV  files, including
-all associations and with configurable defaults.
+all associations and setting configurable defaults or over rides.
 
-Export all data or simply generate a sample template with headers only.
+Generate a sample template with headers only.
+
+Export template and populate with model data 
 
 Create, parse and use Excel/OpenOffice (.xls) documents dynamically from Ruby.
 
@@ -24,7 +26,7 @@ Specific loaders and command line tasks provided out the box for **Spree E-Comme
 enabling import/export of Product data including creating Variants with different
  count on hands and all associations including Properties/Taxons/OptionTypes and Images.
 
-Loaders can be configured via YAML with over ride values, default values and mandatory column settingss.
+Loaders can be configured via YAML with over ride values, default values and mandatory column settings.
 
 Many example Spreadsheets/CSV files in spec/fixtures, fully documented with comments for each column.
 
@@ -41,13 +43,11 @@ To use :
 
 To pull the tasks in, add this call to your Rakefile :
 
-     DataShift::load_tasks
+```ruby DataShift::load_tasks```
 
 To keep the availability to only development mode use
 
-    if(Rails.env.development?)
-        DataShift::<b>load_tasks</b>
-    end
+```ruby DataShift::load_tasks if(Rails.env.development?)```
 
 To use the Thor command line applications :
 
@@ -58,7 +58,7 @@ Edit the file and add the following to pull in the thor commands :
     require 'thor'
     require 'datashift'
 
-    DataShift::<b>load_commands</b>
+    DataShift::load_commands
 ```
 To check the available tasks run
 
@@ -88,6 +88,9 @@ Guards are provided, and used internally, for mixed Ruby setups. Can be used lik
 Provides high level rake tasks for importing data via ActiveRecord models into a DB,
  from various sources, currently csv or .xls files (Excel/Open Office)
 
+N.B This is under active development, moving to thor tasks so maybe out of date
+
+Please try rake -T and  thor list to get upto date command lines
 
     bundle exec rake datashift:import:csv model=BlogPost input=BlogPostImport.csv verbose=true 
 
@@ -102,54 +105,9 @@ The library can be easily extended with Loaders to deal with non trivial cases,
  for example when multiple lookups required to find right association.
 
 Spree loaders are an example, these illustrate over riding processing for specific columns with
-complicated lookup requirements.
-
-A core feature of DataShift is the MethodDictionary and MethodMapper, which provides features for collecting
-reflection information from ActiveRecord models (all different associations, including join tables with many-to-many relationships).
-
-A full picture of all possible operations on a class can be created very easily, for example ona  Blog model :
-
-    MethodDictionary.find_operators( Blog )
-
-This then allows Import/Export to be achieved, by mapping the file's header and column data to the found operators
-i.e. the methods to set data on model's attributes and associations.
-
-Here we retrieve the method details for a column name from a file, "Blog Date"
-
-    MethodDictionary.find_method_detail( Blog, "Blog Date" )
-
-Loaders can use this method lookup functionality, to find the correct association for a column heading,
-and populate AR object with row data.
-
-This means data can be mapped to any model without any further coding. Generators are also supplied to export
-a models attributes and associations to files, thus providing template spreadsheets that any user can fill out.
-
-MethodMapper also stores column type information so the raw file data can be provided as is,
-and whenever possible, under the bonnet the data will be cast to correct DB type.
-
-Here we show how a column name from a file, "Blog Date", can be mapped to Assign a stringified date, to the blog_date column, on a new Blog object :
-
-    MethodDictionary.find_method_detail( Blog, "Blog Date" ).assign( Blog.new, "Sat Jul 23 2011" )
-
-Because it's based on reflection against the model, can build complex relationships between tables during import/export, 
-and extending data files with new columns need not require any additional Ruby coding.
-
-New columns can simply be added to the Excel/Open Office spreadsheet, or CSV file, setting the new
-attribute or association name in the header row. 
-
-
-The Loader attempts to handle various human read-able forms of column names.
-
-For example, given an association on the model called, product_properties, will successfully load
-from columns with headings such as 'product_properties', 'Product Properties', 'ProductProperties' 'product properties' etc
-
-For has_many associations, either multiple columns can be used,
- or multiple values can be specified in a single column using suitable delimiters.
-
-Modular - so complex associations/mappings that require non generic lookups, can be handled by extending the loader engine.
-
-Original focus was on support for the Open Source Spree e-commerce project, so includes specific loaders and rake tasks
-for loading Spree Products, and associated data such as Product Variants, and Images.
+complicated lookup requirements. Spree is the prime Open Source e-commerce project for Rails, 
+and the specific loaders and tasks support loading Spree Products, and associated data such as Variants,
+OptionTypes, Properties and Images.
 
 ## Template Generation and Export
 
@@ -256,48 +214,6 @@ So if our Project model has many Categories, we can supply a Category list, whic
 During loading, a call to find_all_by_reference will be made, picking up the 2 categories with matching references,
  and our Project model will contain those two i.e project.categories = [category_002,category_003]
 
-## Spree Suppprt
-
-### OptionTypes & Variants
-
-When loaded with the Spree specific tasks, spree specific over rides are supported, such as direct s
-support for OptionTypes with values
-
-Any 'Option Types' columns can contain the OptionType to associate with the Product, plus a selection of
-appropriate OptionValues to go with that Type. 
-
-For example, in a single column/row we could supply 2 OptionTypes (named, size & colour), with a selection values
-(such as small, medium etc)
-
-    'Option Types'
-    size:small,medium,large|colour:red,white
-
-If no such OptionType exists, e.g size, then a new one is created with the supplied name.
-
-Next the OptionValues are also parsed, again if no such OptionValue exists, e.g small, then a new one is created with the supplied name.
-
-Lastly a Variant is created on each OptionValue, with price and availaable dates being copied from Master.
-Currently a unique SKU is created by adding an index to the master's sku.
-
-TODO - Enable a hash of attributes to be supplied in association columns to enable more control over creation of associated objects.
-
-### Properties
-
-The properties to associate with this product.
-Properties are for small snippets of text, shared across many products,
-and are for display purposes only.
-
-An optional display value can be supplied to supplement the displayed text.
-
-As for all associations can contain multiple name/value sets in default form :
-
-    Property:display_value|Property:display_value
-
-Example - No values :
-    manufacturer|standard
-
-Example - Display  values :
-    manufacturer:somebody else plc|standard:ISOBlah21
 
 ## TODO
 

data/VERSION

@@ -1 +1 @@
-0.7.0
+0.8.0

data/datashift.gemspec

@@ -5,11 +5,11 @@
 
 Gem::Specification.new do |s|
   s.name = "datashift"
-  s.version = "0.7.0"
+  s.version = "0.8.0"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["Thomas Statter"]
-  s.date = "2012-05-27"
+  s.date = "2012-07-09"
   s.description = "A suite of tools to move data between ActiveRecord models,databases,applications like Excel/Open Office, files and projects including Spree"
   s.email = "rubygems@autotelik.co.uk"
   s.extra_rdoc_files = [
@@ -24,8 +24,6 @@ Gem::Specification.new do |s|
     "README.rdoc",
     "Rakefile",
     "VERSION",
-    "datashift-0.6.0.gem",
-    "datashift-0.6.1.gem",
     "datashift.gemspec",
     "lib/applications/jruby/jexcel_file.rb",
     "lib/applications/jruby/word.rb",
@@ -67,6 +65,7 @@ Gem::Specification.new do |s|
     "lib/loaders/csv_loader.rb",
     "lib/loaders/excel_loader.rb",
     "lib/loaders/loader_base.rb",
+    "lib/loaders/paperclip/image_loader.rb",
     "lib/loaders/spreadsheet_loader.rb",
     "lib/loaders/spree/image_loader.rb",
     "lib/loaders/spree/product_loader.rb",
@@ -75,6 +74,7 @@ Gem::Specification.new do |s|
     "lib/thor/import_excel.thor",
     "lib/thor/spree/bootstrap_cleanup.thor",
     "lib/thor/spree/products_images.thor",
+    "lib/thor/spree/reports.thor",
     "public/spree/products/large/DEMO_001_ror_bag.jpeg",
     "public/spree/products/large/DEMO_002_Powerstation.jpg",
     "public/spree/products/large/DEMO_003_ror_mug.jpeg",
@@ -124,6 +124,7 @@ Gem::Specification.new do |s|
     "spec/fixtures/simple_export_spec.xls",
     "spec/fixtures/simple_template_spec.xls",
     "spec/fixtures/spree/SpreeImages.xls",
+    "spec/fixtures/spree/SpreeMultiVariant.csv",
     "spec/fixtures/spree/SpreeProducts.csv",
     "spec/fixtures/spree/SpreeProducts.xls",
     "spec/fixtures/spree/SpreeProductsDefaults.yml",
@@ -149,6 +150,7 @@ Gem::Specification.new do |s|
     "spec/thor_spec.rb",
     "tasks/config/seed_fu_product_template.erb",
     "tasks/config/tidy_config.txt",
+    "tasks/db_tasks.rake",
     "tasks/export/excel_generator.rake",
     "tasks/file_tasks.rake",
     "tasks/import/csv.rake",

data/lib/applications/jruby/jexcel_file.rb

@@ -25,13 +25,15 @@ if(DataShift::Guards::jruby?)
     include_class 'org.apache.poi.hssf.usermodel.HSSFDataFormat'
     include_class 'org.apache.poi.hssf.usermodel.HSSFClientAnchor'
     include_class 'org.apache.poi.hssf.usermodel.HSSFRichTextString'
+    
+    include_class 'org.apache.poi.hssf.util.HSSFColor'
 
     include_class 'java.io.ByteArrayOutputStream'
     include_class 'java.util.Date'
     include_class 'java.io.FileInputStream'
     include_class 'java.io.FileOutputStream'
 
-    attr_accessor :book, :row, :date_style
+    attr_accessor :workbook, :row, :date_style
     attr_reader   :sheet
 
     MAX_COLUMNS = 256.freeze unless defined?(MAX_COLUMNS)
@@ -45,27 +47,25 @@ if(DataShift::Guards::jruby?)
       return 65535
     end
 
-    # The HSSFWorkbook uses 0 based indexes, whilst our companion jexcel_win32 class
-    # uses 1 based indexes. So they can be used interchangeably we bring indexes 
-    # inline with  JExcel usage in this class, as 1 based maps more intuitively for the user
-    # 
-    # i.e Row 1 passed to this class, internally means Row 0
+    # The HSSFWorkbook uses 0 based indexes, 
   
-    def initialize()
-      @book = nil
+    def initialize(filename = nil)
+      @workbook = nil
       # The @patriarchs hash is a workaround because HSSFSheet.getDrawingPatriarch()
       # causes a lot of issues (if it doesn't throw an exception!)
       @patriarchs = Hash.new
       
       @date_style = nil
+      
+      open(filename) if(filename)
     end
   
     def open(filename)
       inp = FileInputStream.new(filename)
 
-      @book = HSSFWorkbook.new(inp)
+      @workbook = HSSFWorkbook.new(inp)
       
-      @date_style = @book.createCellStyle
+      @date_style = @workbook.createCellStyle
       @date_style.setDataFormat( JExcelFile::date_format )
       
       @current_sheet = 0
@@ -75,19 +75,19 @@ if(DataShift::Guards::jruby?)
     # EXCEL ITEMS
     
     def create(sheet_name)
-      @book = HSSFWorkbook.new() if @book.nil?
+      @workbook = HSSFWorkbook.new() if @workbook.nil?
 
       acceptable_name = sheet_name.gsub(':', '').gsub(" ", '')
       
       # Double check sheet doesn't already exist
-      if(@book.getSheetIndex(acceptable_name) < 0)
-        sheet = @book.createSheet(acceptable_name.gsub(" ", ''))
+      if(@workbook.getSheetIndex(acceptable_name) < 0)
+        sheet = @workbook.createSheet(acceptable_name.gsub(" ", ''))
 
         @patriarchs.store(acceptable_name, sheet.createDrawingPatriarch())
       end
-      @current_sheet = @book.getSheetIndex(acceptable_name)
+      @current_sheet = @workbook.getSheetIndex(acceptable_name)
       
-      @date_style = @book.createCellStyle
+      @date_style = @workbook.createCellStyle
       @date_style.setDataFormat( JExcelFile::date_format )
       
       self.sheet()
@@ -98,18 +98,18 @@ if(DataShift::Guards::jruby?)
     # Return the current or specified HSSFSheet
     def sheet(i = nil)
       @current_sheet = i if i
-      @sheet = @book.getSheetAt(@current_sheet)
+      @sheet = @workbook.getSheetAt(@current_sheet)
     end
 
     def activate_sheet(sheet)
       active_sheet = @current_sheet
-      if(@book)
+      if(@workbook)
         i = sheet if sheet.kind_of?(Integer)
-        i = @book.getSheetIndex(sheet) if sheet.kind_of?(String)
+        i = @workbook.getSheetIndex(sheet) if sheet.kind_of?(String)
 
         if( i >= 0 )
-          @book.setActiveSheet(i) unless @book.nil?
-          active_sheet = @book.getSheetAt(i)
+          @workbook.setActiveSheet(i) unless @workbook.nil?
+          active_sheet = @workbook.getSheetAt(i)
           active_sheet.setActive(true)
         end unless i.nil?
       end
@@ -126,11 +126,11 @@ if(DataShift::Guards::jruby?)
       @sheet.rowIterator.each { |row| @row = row; yield row }
     end
 
-    # Create new row, bring index in line with POI usage (our 1 is their 0)
+    # Create new row,  index with POI usage starts at 0
     def create_row(index)
       return if @sheet.nil?
-      raise "BAD INDEX: Row indexing starts at 1" if(index == 0)
-      @row = @sheet.createRow(index - 1)
+      raise "BAD INDEX: Row indexing starts at 0" if(index < 0)
+      @row = @sheet.createRow(index)
       @row
     end
     
@@ -141,20 +141,34 @@ if(DataShift::Guards::jruby?)
     #  Populate a single cell with data
     #    
     def set_cell(row, column, datum)
-      @row = @sheet.getRow(row - 1) || create_row(row)
-      @row.createCell(column - 1, excel_cell_type(datum)).setCellValue(datum)
+      @row = @sheet.getRow(row) || create_row(row)
+      @row.createCell(column, excel_cell_type(datum)).setCellValue(datum)
     end
     
     # Convert array into a header row
-    def set_headers(headers)
-      create_row(1)
+    def set_headers(headers, apply_style = nil)
+      create_row(0)
       return if headers.empty?
     
+      style = apply_style || header_style()
+
       headers.each_with_index do |datum, i|
-        @row.createCell(i, excel_cell_type(datum)).setCellValue(datum)
+        c = @row.createCell(i, excel_cell_type(datum))
+        c.setCellValue(datum)
+        c.setCellStyle(style)
       end
     end
 
+    def header_style
+      return @header_style if @header_style
+      @header_style = @workbook.createCellStyle();
+      @header_style.setBorderTop(6) # double lines border
+      @header_style.setBorderBottom(1) # single line border
+      @header_style.setFillBackgroundColor(HSSFColor::GREY_25_PERCENT.index)
+      
+      @header_style
+    end
+    
     #  Populate a row  of cells with data in an array 
     #  where the co-ordinates relate to row/column start position
     #    
@@ -171,7 +185,8 @@ if(DataShift::Guards::jruby?)
       end
     end
   
-    # Return a mapping from Ruby type to type for HSSFCell
+    # Return the suitable type for a HSSFCell from a Ruby data type
+    
     def excel_cell_type(data)
         
       if(data.kind_of?(Numeric))
@@ -186,6 +201,17 @@ if(DataShift::Guards::jruby?)
       # HSSFCell::CELL_TYPE_FORMULA
     end
     
+    
+    # Auto size either the given column index or all columns
+    def autosize(column = nil)
+      return if @sheet.nil?
+      if (column.kind_of? Integer)
+        @sheet.autoSizeColumn(column)
+      else
+        @sheet.getRow(0).cellIterator.each{|c| @sheet.autoSizeColumn(c.getColumnIndex)}
+      end
+    end
+    
     # TODO - Move into an ActiveRecord helper module of it's own
     def ar_to_headers( records )
       return if( !records.first.is_a?(ActiveRecord::Base) || records.empty?)
@@ -193,23 +219,24 @@ if(DataShift::Guards::jruby?)
       headers = records.first.class.columns.collect( &:name )    
       set_headers( headers )
     end
-        
+      
+    
     # Pass a set of AR records
     def ar_to_xls(records, options = {})
       return if( ! records.first.is_a?(ActiveRecord::Base) || records.empty?)
       
       row_index = 
         if(options[:no_headers])
-        1
+        0
       else
         ar_to_headers( records )
-        2
+        1
       end
       
       records.each do |record|
         create_row(row_index)
  
-        ar_to_xls_row(1, record)
+        ar_to_xls_row(0, record)
         
         row_index += 1
       end
@@ -233,15 +260,15 @@ if(DataShift::Guards::jruby?)
         datum = record.send(connection_column.name)
 
         if(connection_column.sql_type =~ /date/) 
-          @row.createCell(column - 1, HSSFCell::CELL_TYPE_STRING).setCellValue(datum.to_s) 
+          @row.createCell(column, HSSFCell::CELL_TYPE_STRING).setCellValue(datum.to_s) 
           
         elsif(connection_column.type == :boolean || connection_column.sql_type =~ /tinyint/) 
-          @row.createCell(column - 1, HSSFCell::CELL_TYPE_BOOLEAN).setCellValue(datum) 
+          @row.createCell(column, HSSFCell::CELL_TYPE_BOOLEAN).setCellValue(datum) 
           
         elsif(connection_column.sql_type =~ /int/) 
-          @row.createCell(column - 1, HSSFCell::CELL_TYPE_NUMERIC).setCellValue(datum.to_i)
+          @row.createCell(column, HSSFCell::CELL_TYPE_NUMERIC).setCellValue(datum.to_i)
         else
-          @row.createCell(column - 1, HSSFCell::CELL_TYPE_STRING).setCellValue( datum.to_s ) 
+          @row.createCell(column, HSSFCell::CELL_TYPE_STRING).setCellValue( datum.to_s ) 
         end
         
       rescue => e
@@ -263,7 +290,6 @@ if(DataShift::Guards::jruby?)
     # Return the raw data of an HSSFCell
     def cell_value(cell)
       return unless cell
-      #puts "DEBUG CELL TYPE : #{cell} => #{cell.getCellType().inspect}"
       case (cell.getCellType())
       when HSSFCell::CELL_TYPE_FORMULA  then return cell.getCellFormula()
       when HSSFCell::CELL_TYPE_NUMERIC  then return cell.getNumericCellValue()
@@ -278,7 +304,7 @@ if(DataShift::Guards::jruby?)
       filename.nil? ? file = @filepath : file = filename
       begin
         out = FileOutputStream.new(file)
-        @book.write(out) unless @book.nil?
+        @workbook.write(out) unless @workbook.nil?
 
         out.close
       rescue => e
@@ -312,28 +338,19 @@ if(DataShift::Guards::jruby?)
   
     # Get a percentage style
     def getPercentStyle()
-      if (@percentCellStyle.nil? && @book)
-        @percentCellStyle = @book.createCellStyle();
+      if (@percentCellStyle.nil? && @workbook)
+        @percentCellStyle = @workbook.createCellStyle();
         @percentCellStyle.setDataFormat(HSSFDataFormat.getBuiltinFormat("0.00%"));
       end
       return @percentCellStyle
     end
 
-    # Auto size either the given column index or all columns
-    def autosize(column = nil)
-      return if @sheet.nil?
-      if (column.kind_of? Integer)
-        @sheet.autoSizeColumn(column)
-      else
-        @sheet.getRow(0).cellIterator.each{|c| @sheet.autoSizeColumn(c.getColumnIndex)}
-      end
-    end
 
     def to_s
-      return "" unless @book
+      return "" unless @workbook
             
       outs = ByteArrayOutputStream.new
-      @book.write(outs);
+      @workbook.write(outs);
       outs.close();
       String.from_java_bytes(outs.toByteArray)
     end