datashift 0.15.0 → 0.16.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 4e73de99a2466b06c8dfb6bab01cb7c2ab00fb54
+  data.tar.gz: 73c47634a567004886c55e75b1f03a63a75ff155
+SHA512:
+  metadata.gz: ec2035080b4d6cfc1560530639084e35271f2716a6772f0b8ada9d2d62752f8c3e5756babc4e5bc9afa83636b9adef4c047bc5db99ad26e31e8e75e51ff3b89d
+  data.tar.gz: 426c72aadaab70fa3e1a09c8b131a0b862e75d94ad138316474cf75cf9af0a27aaf7d72037f85404d94dbdab7c668b8a47f5939da56914d0c6591421fd79fd4a

data/README.markdown

@@ -1,59 +1,20 @@
 ##  DataShift 
 
+- [Installation](#Installation)
 - [Features](#features)
-- [Installation](#installation)
-- [Active Record - Import/Export](#Active Record - Import/Export)
+- [Testing](#testing)
 - [License](#license)
 
-Provides tools to shift data between Excel/CSV files and Rails projects and Ruby applications
-
-Import and export models fully with all associations.
+Shift data between Excel/CSV files and Rails or Ruby applications
 
 Comprehensive Wiki here : **https://github.com/autotelik/datashift/wiki**
 
-Specific command line tools and full Product loading for Spree E-Commerce 
-now separate gem at [datashift_spree](https://github.com/autotelik/datashift_spree "Datashift Spree")
-
-
-### Features
-
-Import and Export ActiveRecord models direct to CSV or Excel/OpenOffice (.xls) (JRuby, 1.8.7, REE, 1.9.3)
-
-You can select which associations to include and for import, set configurable defaults or over rides.
-
-Create, parse and use Excel/OpenOffice (.xls) documents dynamically from Ruby (JRuby, 1.8.7, REE, 1.9.3)
-
-Generate a sample template with headers only.
-
-Export template and populate with model data 
-
-Bulk import tools for Paperclip attachments.
-
-Easily extendable Loader functionality to deal with non trivial import cases, such
-as complex association lookups.
-
-High level rake and thor command line tasks for import/export provided.
-
-Specific loaders and command line tasks provided out the box for **Spree E-Commerce**, 
-enabling import/export of Product data including creating Variants with different
- count on hands and all associations including Properties/Taxons/OptionTypes and Images.
-
-Loaders can be configured via YAML with over ride values, default values and mandatory column settings.
-
-Many example Spreadsheets/CSV files in spec/fixtures, fully documented with comments for each column.
-
-## Installation
+### <a name="Installation">Installation</a>
 
 Add gem 'datashift' to your Gemfile/bundle or use ```gem install```
 
-```ruby 
-gem 'datashift' 
-```
-
-For Spree support also add :
-
-```ruby 
-gem 'datashift_spree'
+```ruby
+gem 'datashift'
 ```
 
 To use :
@@ -64,7 +25,7 @@ To use the Thor command line applications, pull in the tasks.
 
 Generally the easiest way is to, create a high level .thor file in your Rails root directory
 
-    e.g mysite.thor  
+    e.g mysite.thor
 
 Edit the file and add the following to pull in the thor commands :
 
@@ -89,6 +50,33 @@ To get usage information use thor help <command>, for example
 
 To use Excel OLE and MS Excel are NOT required.
 
+Specific tools for Spree E-Commerce now separate gem [datashift_spree](https://github.com/autotelik/datashift_spree "Datashift Spree")
+
+
+#### <a name="Features">Features</a>
+
+Import and Export ActiveRecord models direct to CSV or Excel/OpenOffice (.xls)
+
+You can select which associations to include and for import, set configurable defaults or over rides.
+
+Create, parse and use Excel/OpenOffice (.xls) documents dynamically from Ruby
+
+Generate a sample template with headers only.
+
+Export template and populate with model data 
+
+Bulk import tools for Paperclip attachments.
+
+Easily extendable Loader functionality to deal with non trivial import cases, such
+as complex association lookups.
+
+High level rake and thor command line tasks for import/export provided.
+
+
+Loaders can be configured via YAML with over ride values, default values and mandatory column settings.
+
+Many example Spreadsheets/CSV files in spec/fixtures, fully documented with comments for each column.
+
 Features a common Excel interface over both our own wrapper around Apache POI (JRuby) and spreadsheet gem (all main Rubies) 
 
 This means you can switch seamlessly between the two libraries, and if required drop down to make use of advanced
@@ -102,7 +90,7 @@ Guards are provided, and used internally, for mixed Ruby setups. Can be used lik
         ..do something with speadsheet
     end
 
-## Active Record - Import/Export
+#### Active Record - Import/Export
 
 Provides high level tasks for importing data via ActiveRecord models into a DB,
  from various sources, currently csv or .xls files (Excel/Open Office)
@@ -145,7 +133,7 @@ complicated lookup requirements. Spree is the prime Open Source e-commerce proje
 and the specific loaders and tasks support loading Spree Products, and associated data such as Variants,
 OptionTypes, Properties and Images.
 
-## Template Generation and Export
+#### Template Generation and Export
 
 Template generation tasks can be used to export a model's definition as column headings to CSV or .xls.
 These can be provided to developers or business users, as a template for data collection and then loading.
@@ -155,7 +143,7 @@ Export tasks can be used to export of a model's definition and any existing data
 This data can be exported directly to CSV or Excel/OpenOffice spreadsheets.
 
 
-## Example Spreadsheets
+#### Example Spreadsheets
     
   A number of example Spreadsheets with headers and comments, can be found in the spec/fixtures directory.
 
@@ -165,7 +153,7 @@ This data can be exported directly to CSV or Excel/OpenOffice spreadsheets.
   Column headings contain comments with full descriptions and instructions on syntax. 
 
 
-## Excel
+#### Excel
 
 
   MS Excel itself does not need to be installed.
@@ -181,7 +169,7 @@ This data can be exported directly to CSV or Excel/OpenOffice spreadsheets.
   without converting first to CSV or YAML.
 
 
-### Associations
+#### Associations
 
 To perform a lookup for an associated model, the primary column(s) must be supplied, along with required select values for those columns.
 
@@ -199,7 +187,7 @@ During loading, a call to find_all_by_reference will be made, picking up the 2 c
  and our Project model will contain those two i.e project.categories = [category_002,category_003]
 
 
-## TODO
+### TODO
 
   - Smart sorting of column processing order ....
 
@@ -208,14 +196,62 @@ During loading, a call to find_all_by_reference will be made, picking up the 2 c
   - Look at implementing import/export API using something like https://github.com/ianwhite/orm_adapter 
     rather than active record, so we can support additional ORMs
 
-    
+
+### <a name="Testing">Testing</a>
+    Specs have own Gemfile, so you can specify versions of active record that you want  specs to run against :
+
+    Edit
+        ```ruby spec/Gemfile. ```
+
+    Then run :
+
+    ```ruby
+    cd spec
+    bundle install
+    ```
+
+####  Changing Versions
+
+    A sandbox will be generated in spec/sandbox if no such directory exists.
+
+    **N.B Manual Step**
+    When changing versions you probably need to **delete this whole directory**  spec/sandbox. Next time you run spree specs it will be auto generated using latest Rails versions
+
+    The database are created in sqlite3 and are stored in spec/fixtures. When switching versions, of say Spree,
+     you will probably want to and to clear out old versions and retrigger the migrations
+
+        rm spec/fixtures/*.sqlite
+
+    You will probably also want to remove lock file :
+
+        rm spec/Gemfile.lock
+
+    First time the sandbox is regenerated, alot of tests may fail,perhaps not everything loads correctly during regeneration process.
+
+    Invariably the next run, the specs pass, so a fix is low priority.
+
+#### Run the Tests
+
+    ** N.B You should run the specs from within the specs directory. **
+    ```ruby
+        bundle exec rspec -c .
+    ```
+
+    A datashift **log **will be written within **spec/logs**, which hooks into the standard active record logger
+
+          /log/datashift.log
+          spec/logs/datashift_spec.log
+
+
+
+
 ## License
 
-Copyright:: (c) Autotelik Media Ltd 2011
+Copyright:: (c) Autotelik Media Ltd 2015
 
 Author ::   Tom Statter
 
-Date ::     Dec 2011
+Date ::     Dec 2015
 
 The MIT License
 

data/VERSION

@@ -1 +1 @@
-0.15.0
+0.16.0

data/datashift.gemspec

@@ -1,21 +1,16 @@
 require 'rake'
 
-lib = File.expand_path('../lib/', __FILE__)
-
-$:.unshift lib unless $:.include?(lib)
-
-require 'datashift'
+#TODO version = File.read("VERSION").strip
   
 Gem::Specification.new do |s|
   s.name = "datashift"
-  s.version = DataShift::gem_version
-  s.date = Date.today.to_s
+  s.version = "0.16.0"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   
   s.authors = ["Thomas Statter"]
 
-  s.description = "Comprehensive tools to import/export between Excel/CSV and ActiveRecord databases, Rails apps, and any Ruby project."
+  s.description = "Comprehensive import/export tools between Excel/CSV & ActiveRecord Databases, Rails apps, and any Ruby project."
   s.email = "rubygems@autotelik.co.uk"
   s.extra_rdoc_files = [
     "LICENSE.txt",
@@ -42,22 +37,12 @@ Gem::Specification.new do |s|
   s.homepage = "http://github.com/autotelik/datashift"
   s.licenses = ["MIT"]
   s.require_paths = ["lib"]
-  s.rubygems_version = "1.8.24"
+
   s.summary = "Shift data betwen Excel/CSV and any Ruby app"
 
-  if s.respond_to? :specification_version then
-    s.specification_version = 3
-
-    if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
-      s.add_runtime_dependency(%q<spreadsheet>, [">= 0"])
-      s.add_runtime_dependency(%q<rubyzip>, [">= 0"])
-    else
-      s.add_dependency(%q<spreadsheet>, [">= 0"])
-      s.add_dependency(%q<rubyzip>, [">= 0"])
-    end
-  else
-    s.add_dependency(%q<spreadsheet>, [">= 0"])
-    s.add_dependency(%q<rubyzip>, [">= 0"])
-  end
+  s.add_dependency 'spreadsheet'
+  s.add_dependency 'rubyzip'
+
+
 end
 

data/lib/applications/jexcel_file.rb

@@ -97,8 +97,7 @@ if(DataShift::Guards::jruby?)
       else 
         
         name = sanitize_sheet_name( sheet_name )
-        
-        puts "WTF #{name}"
+
         if (@workbook.getSheetIndex(name) < 0)  #Check sheet doesn't already exist
           return create_sheet_and_set_styles( name )
         else

data/lib/datashift.rb

@@ -25,7 +25,7 @@
 
 
 # Details::   Active Record Import/Export for .xls or CSV
-# 
+#
 # To pull DataShift commands into your main application :
 #
 #     require 'datashift'
@@ -58,9 +58,9 @@ module DataShift
   def self.library_path
     File.expand_path("#{File.dirname(__FILE__)}/../lib")
   end
-  
+
   def self.require_libraries
-    
+
     loader_libs = %w{ lib  }
 
     # Base search paths - these will be searched recursively
@@ -78,8 +78,8 @@ module DataShift
         end
       end
     end
-    
-    require_libs = %w{ datashift loaders helpers }
+
+    require_libs = %w{ datashift loaders generators helpers }
 
     require_libs.each do |base|
       Dir[File.join(library_path, base, '*.rb')].each do |rb|
@@ -90,6 +90,15 @@ module DataShift
       end
     end
 
+    if(DataShift::Guards.jruby?)
+      require 'jexcel_file'
+      JExcelFile
+    else
+      require 'spreadsheet'
+      require 'spreadsheet_extensions'
+      Spreadsheet
+    end
+
   end
 
   # Load all the datashift rake tasks and make them available throughout app
@@ -100,36 +109,46 @@ module DataShift
     Dir["#{base}/*.rake"].sort.each { |ext| load ext }
   end
 
-  
+
   # Load all the datashift Thor commands and make them available throughout app
 
   def self.load_commands()
     base = File.join(library_path, 'thor', '**')
-    
+
     Dir["#{base}/*.thor"].each do |f|
       next unless File.file?(f)
       Thor::Util.load_thorfile(f)
     end
   end
-  
+
 end
 
+
+require_relative 'datashift/delimiters'
+require_relative 'datashift/column_packer'
+require_relative 'datashift/logging'
+require_relative 'datashift/exceptions'
+require_relative 'datashift/guards'
+
+require_relative 'helpers/core_ext/to_b'
+require_relative 'helpers/core_ext/csv_file'
+
+require_relative 'datashift/method_detail'
+require_relative 'datashift/method_dictionary'
+require_relative 'datashift/method_mapper'
+require_relative 'datashift/model_mapper'
+
 DataShift::require_libraries
 
-require 'datashift/guards'
-require 'datashift/logging'
-require 'datashift/method_detail'
-require 'datashift/method_dictionary'
-require 'datashift/method_mapper'
 
 module DataShift
   if(Guards::jruby?)
     require 'java'
-      
+
     class Object
       def add_to_classpath(path)
         $CLASSPATH << File.join( DataShift.root_path, 'lib', path.gsub("\\", "/") )
       end
     end
   end
-end
+end

data/lib/datashift/column_packer.rb

@@ -5,60 +5,124 @@
 #
 # Details::   Helper for creating consistent import/export format
 #             of model's attributes/associations
-# 
 #
-require 'exporter_base'
-require 'csv'
-
 module DataShift
 
   module ColumnPacker
 
-    
-    def text_delim
-      @text_delim ||= "\'"
-    end
+    include Delimiters
 
-    def text_delim=(x)
-      @text_delim = x
-    end
-    
     # Return opposite of text delim - "hello, 'barry'" => '"hello, "barry""'
     def escape_text_delim
       return '"' if text_delim == "\'"
       "\'"
     end
-    
- 
+
+
+    # Ensure a value is written to CSV correctly
+    # TODO - better ways ?? - see transcoding and String#encode
+
+    def escape_for_csv(value)
+      text = value.to_s.gsub(text_delim, escape_text_delim()).gsub("\n", "\\n")
+
+      text = "#{text_delim}#{text}#{text_delim}" if(text.include?(Delimiters::csv_delim))
+      text
+    end
+
+
+    def to_headers( records, associations = nil, options = {} )
+      return if( !records.first.is_a?(ActiveRecord::Base) || records.empty?)
+
+      only = *options[:only] ? [*options[:only]] : nil
+
+      headers =[]
+
+      if associations
+        details_mgr = DataShift::MethodDictionary.method_details_mgrs[records.first.class]
+
+        [*associations].each do |a|
+
+          details_mgr.get_list(a).each do  |md|
+
+            next if(only && !only.include?( md.name.to_sym ) )
+
+            headers << "#{md.operator}"
+
+          end
+        end if(details_mgr)
+
+      else
+
+        headers = records.first.class.columns.collect( &:name )
+      end
+
+      headers
+    end
+
+
     # Convert an AR instance to a single column
-  
-    def record_to_column(record, attribute_delim = Delimiters::csv_delim) 
-    
-      csv_data = []
-      record.serializable_hash.each do |name, value|
-        value = 'nil' if value.nil?
-        text = value.to_s.gsub(@text_delim, escape_text_delim())
-        csv_data << "#{name.to_sym} => #{text}"
+    #    e.g User  :  ":name = > 'tom', :role => 'developer'"
+    #
+    # OPTIONS
+    #     with_only  Specify (as symbols) columns for association types to export
+    #     json:         Export association data in single column in JSON format
+    #
+    def record_to_column(record, options = {})
+
+      return "" if(record.nil? || (record.respond_to?(:each) && record.empty?) )
+
+      with_only = *options[:with_only] ? [*options[:with_only]] : nil
+
+      return record.to_json if(options[:json] && !with_only) # packs associations into single column
+
+      if( record.respond_to?(:each) )
+
+        return "" if(record.empty?)
+
+        data = []
+
+        record.each { |r| data << record_to_column(r, options); }
+
+        if(options[:json])
+          return data.to_json
+        else
+          return "#{data.join(Delimiters::multi_assoc_delim)}"
+        end
+
+      else
+
+        data = options[:json] ? {} : []
+
+        record.serializable_hash.each do |name, value|
+          next if(with_only && !with_only.include?( name.to_sym ) )
+
+          if(options[:json])
+            data[name] = value
+          else
+            data << "#{name.to_sym} #{Delimiters::key_value_sep} #{value.to_s.gsub(text_delim, escape_text_delim)}"
+          end
+        end
+
+        if(options[:json])#
+          return data.to_json
+        else
+          "#{Delimiters::attribute_list_start}#{data.join(Delimiters::multi_value_delim)}#{Delimiters::attribute_list_end}"
+        end
+
       end
-      "#{csv_data.join(attribute_delim)}"
+
     end
-    
-    
+
+
     # Convert an AR instance to a set of CSV columns
     def record_to_csv(record, options = {})
       csv_data = record.serializable_hash.values.collect { |value| escape_for_csv(value) }
 
       [*options[:methods]].each { |x| csv_data << escape_for_csv(record.send(x)) if(record.respond_to?(x)) } if(options[:methods])
-      
+
       csv_data.join( Delimiters::csv_delim )
     end
-    
-    def escape_for_csv(value)
-      text = value.to_s.gsub(@text_delim, escape_text_delim())
-      
-      text = "#{@text_delim}#{text}#{@text_delim}" if(text.include?(Delimiters::csv_delim)) 
-      text
-    end
-    
+
+
   end
 end