datashift 0.9.0 → 0.10.0

data/README.markdown

@@ -1,21 +1,27 @@
 ##  DataShift 
 
-Provides tools to shift data between Ruby, external applications, files and ActiveRecord.
+Provides tools to shift data between Excel/CSV files and Rails projects and Ruby applications
 
-Specific loaders and command line tasks for Spree E-Commerce.
+Import and export models fully with all associations.
 
-Wiki taking shape with more info here : **https://github.com/autotelik/datashift/wiki**
+Comprehensive Wiki here : **https://github.com/autotelik/datashift/wiki**
+
+Specific command line tools and full Product loading for Spree E-Commerce 
+now seperate gem at [datashift_spree](https://github.com/autotelik/datashift_spree "Datashift Spree")
 
 ### Features
 
-Import and Export ActiveRecord models through .xls or CSV  files, including
-all associations and setting configurable defaults or over rides.
+Import and Export ActiveRecord models direct to CSV or Excel/OpenOffice (.xls) (JRuby, 1.8.7, REE, 1.9.3)
+
+You can select which associations to include and for import, set configurable defaults or over rides.
+
+Create, parse and use Excel/OpenOffice (.xls) documents dynamically from Ruby (JRuby, 1.8.7, REE, 1.9.3)
 
 Generate a sample template with headers only.
 
 Export template and populate with model data 
 
-Create, parse and use Excel/OpenOffice (.xls) documents dynamically from Ruby.
+Bulk import tools for Paperclip attachments.
 
 Easily extendable Loader functionality to deal with non trivial import cases, such
 as complex association lookups.
@@ -32,85 +38,73 @@ Many example Spreadsheets/CSV files in spec/fixtures, fully documented with comm
 
 ## Installation
 
-Add gem 'datashift' to your Gemfile/bundle, or install the latest gem as usual :
+Add gem 'datashift' to your Gemfile/bundle or use ```gem install```
 
-    gem install datashift
-
-To use :
+    ```ruby gem 'datashift' ```
 
-    gem 'datashift'
-    require 'datashift'
-
-To pull the tasks in, add this call to your Rakefile :
-
-<<<<<<< HEAD
-```ruby 
-    DataShift::load_tasks
-```
+For Spree support also add :
 
-To keep the availability to only development mode use
+    ```ruby gem 'datashift_spree' ```
 
-```ruby 
-    DataShift::load_tasks if(Rails.env.development?)
-```
-=======
-```ruby DataShift::load_tasks```
+To use :
 
-To keep the availability to only development mode use
+    require 'datashift'
 
-```ruby DataShift::load_tasks if(Rails.env.development?)```
->>>>>>> a6d5d492fd9df38a4acb8c50423219994b4b5601
+To use the Thor command line applications, pull in the tasks.
 
-To use the Thor command line applications :
+Generally the easiest way is to, create a high level .thor file in your Rails root directory
 
-    Create a high level .thor file - e.g mysite.thor - in your applications root directory 
+    e.g mysite.thor  
 
 Edit the file and add the following to pull in the thor commands :
+
 ```ruby
     require 'thor'
     require 'datashift'
 
     DataShift::load_commands
 ```
-To check the available tasks run
 
-    bundle exec rake -T datashift
+To keep the availability to only development mode use
 
-and/or
+```ruby DataShift::load_commands if(Rails.env.development?) ```
 
-    bundle exc thor list datashift
+To check the available tasks run
+
+    bundle exec thor list datashift
 
 To get usage information use thor help <command>, for example
 
     bundle exec thor help datashift:generate:excel
 
-N.B - To use the Excel loader, OLE and Excel are NOT required, however
-JRuby is required, since it uses Java's Apache POI under the hood to process .xls files.
+To use Excel OLE and MS Excel are NOT required.
+
+Features a common Excel interface over both our own wrapper around Apache POI (JRuby) and spreadsheet gem (all main Rubies) 
+
+This means you can switch seamlessly between the two libraries, and if required drop down to make use of advanced
+features in the brilliant Apache POI libraries for anyone using JRuby.
 
 Guards are provided, and used internally, for mixed Ruby setups. Can be used like :
 
     if(DataShift::Guards::jruby? )
-        ..do something with Excel
+        ..do something with Apache
     else
-        ..do something with CSV
+        ..do something with speadsheet
     end
 
 ## Active Record - Import/Export
 
-Provides high level rake tasks for importing data via ActiveRecord models into a DB,
+Provides high level tasks for importing data via ActiveRecord models into a DB,
  from various sources, currently csv or .xls files (Excel/Open Office)
 
-N.B This is under active development, moving to thor tasks so maybe out of date
+Please use thor list and thor help <xxx> to get latest command lines
 
-Please try rake -T and  thor list to get upto date command lines
+    bundle exec thor datashift:import:csv model=BlogPost input=BlogPostImport.csv verbose=true 
 
-    bundle exec rake datashift:import:csv model=BlogPost input=BlogPostImport.csv verbose=true 
 
-    jruby -S rake datashift:import:excel model=BlogPost input=BlogPostImport.xls verbose=true 
+Provides high level  tasks for exporting data to various sources, currently .xls files (Excel/Open Office)
 
-Provides high level rake tasks for exporting data to various sources, currently .xls files (Excel/Open Office)
-
-    jruby -S rake datashift:export:excel model=BlogPost result=BlogExport.xls 
+    bundle exec thor datashift:export:excel model=BlogPost result=BlogExport.xls 
 
 
 The library can be easily extended with Loaders to deal with non trivial cases,
@@ -143,13 +137,27 @@ This data can be exported directly to CSV or Excel/OpenOffice spreadsheets.
 
 ## Features
 
+- *Associations*
+
+  Can import/export 'belongs_to, 'has_many' and 'has_one' associations, including assignment of multiple objects
+  via either multiple columns, or via a DSL for creating multiple entries in a single (column). 
+
+  The DSL can also be used to define which fields to lookup associations, and assign values to other fields.
+
+  See Wiki for more details on DSL syntax.
+
+  Supports delegated attributes.
+
 - *High level wrappers around applications including Excel and Word
 
   Quickly and easily access common enterprise applications through Ruby
 
-  Wrapper around MS Excel File format, acheived via Apache POI under JRuby, so not restricted to Windows
-  and Excel does not need to be installed.
+  MS Excel itself does not need to be installed.
 
+  Our proxy for Excel allows seamless switching between 'spreadsheet' gem and datashift's own JRuby wrapper over Apache POI.
+
+  When using JRuby, Apache POI may offer advanced facilities not found in standard Ruby spreadsheet gem
+  
   The required POI jars are already included.
 
 - *Direct Excel export*
@@ -168,26 +176,21 @@ This data can be exported directly to CSV or Excel/OpenOffice spreadsheets.
   an approximation of the actual associations, so given 'Product Properties' heading,
   finds real association 'product_properties' to send or call on the AR object
 
-- *Associations*
-
-  Can handle 'belongs_to, 'has_many' and 'has_one' associations, including assignment of multiple objects
-  via either multiple columns, or via specially delimited entry in a single (column). See Details section.
 
-  Supports delegated attributes.
 
-- *Rake Tasks*
+- *Thor Tasks*
 
-  High level Rake tasks are provided, only required to supply model class, and file location :
+  High level Thor CLIs are provided, only required to supply model class, and file location :
 
-    jruby -S rake datashift:import:excel model=MusicTrack input=MyTrackListing.xls
+    thor datashift:import:excel model=MusicTrack input=MyTrackListing.xls
 
 
-- *Spree Rake Tasks*
+- *Spree Tasks*
 
   Spree's product associations are non trivial so specific Rake tasks are also provided for loading Spree Producta
   with all associations and Image loading.
 
-    jruby -S rake datashift:spree:products input=C:\MyProducts.xls
+    thor datashift:spree:products input=C:\MyProducts.xls
 
 
 - *Seamless Spree Image loading can be achieved by ensuring SKU or class Name features in Image filename.
@@ -198,14 +201,12 @@ This data can be exported directly to CSV or Excel/OpenOffice spreadsheets.
   parameter klass=Xyz.
 
   In the Spree tasks, this defaults to Product, so attempts to attach Image to a Product via Product SKU or Name.
- 
-  Image loading **does not** specifically require JRuby
 
   A report is generated in the current working directory detailing any Images in the paths that could not be matched with a Product.
 
-  rake  datashift:spree:images input=C:\images\product_images skip_if_no_assoc=true
+    thor  datashift:spree:images input=C:\images\product_images skip_if_no_assoc=true
 
-  rake  datashift:spree:images input=C:\images\taxon_icons skip_if_no_assoc=true klass=Taxon
+    thor datashift:spree:images input=C:\images\taxon_icons skip_if_no_assoc=true klass=Taxon
 
 ## Import to Active Record
 
@@ -229,8 +230,6 @@ During loading, a call to find_all_by_reference will be made, picking up the 2 c
 
 ## TODO
 
-  - Add direct Image loading to Spree  i.e should be able to specify just path in a column .. "/images/red-tshirt.jpg"
-
   - Smart sorting of column processing order ....
 
   - Does not currently ensure mandatory columns (for valid?) processed first.

data/Rakefile

@@ -10,14 +10,11 @@
 #             type applications, files and databases.
 #
 #             Provides support for moving data between .xls (Excel/OpenOffice)
-#             Spreedsheets via JRuby and AR, enabling direct import/export of 
+#             Spreedsheets via Ruby and AR, enabling direct import/export of 
 #             ActiveRecord models with all their associations from database.
 #
 #             Provides support for moving data between csv files and AR, enabling direct
 #             import/export of AR models and their associations from database.
-#             
-#             Provides rake tasks specifically tailored for uploading or exporting
-#             Spree Products, associations and Images
 #
 ## encoding: utf-8
 
@@ -33,8 +30,8 @@ Jeweler::Tasks.new do |gem|
   gem.name = DataShift.gem_name
   gem.homepage = "http://github.com/autotelik/datashift"
   gem.license = "MIT"
-  gem.summary = %Q{ Shift data betwen applications and Active Record}
-  gem.description = %Q{Comprehensive Excel and CSV import/export tools. Shift data between ActiveRecord databases, applications, and projects like Spree}
+  gem.summary = %Q{ Shift data betwen Excel/CSV and any Ruby app}
+  gem.description = %Q{ Comprehensive tools to import/export between Excel/CSV and ActiveRecord databases, Rails apps, and any Ruby projec.}
   gem.email = "rubygems@autotelik.co.uk"
   gem.authors = ["Thomas Statter"]
   # dependencies defined in Gemfile
@@ -89,4 +86,4 @@ task :build, :version do |t, args|
 end
   
 # Long parameter lists so ensure rake -T produces nice wide output
-ENV['RAKE_COLUMNS'] = '180'
+ENV['RAKE_COLUMNS'] = '180'

data/VERSION

@@ -1 +1 @@
-0.9.0
+0.10.0

data/datashift.gemspec

@@ -5,12 +5,12 @@
 
 Gem::Specification.new do |s|
   s.name = "datashift"
-  s.version = "0.9.0"
+  s.version = "0.10.0"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["Thomas Statter"]
-  s.date = "2012-09-03"
-  s.description = "Comprehensive Excel and CSV import/export tools. Shift data between ActiveRecord databases, applications, and projects like Spree"
+  s.date = "2012-09-25"
+  s.description = " Comprehensive tools to import/export between Excel/CSV and ActiveRecord databases, Rails apps, and any Ruby projec."
   s.email = "rubygems@autotelik.co.uk"
   s.extra_rdoc_files = [
     "LICENSE.txt",
@@ -25,8 +25,15 @@ Gem::Specification.new do |s|
     "Rakefile",
     "VERSION",
     "datashift.gemspec",
-    "lib/applications/jruby/jexcel_file.rb",
+    "lib/applications/apache_poi_extensions.rb",
+    "lib/applications/excel.rb",
+    "lib/applications/excel_base.rb",
+    "lib/applications/jexcel_file.rb",
+    "lib/applications/jexcel_file_extensions.rb",
+    "lib/applications/jruby/old_pre_proxy_jexcel_file.rb",
     "lib/applications/jruby/word.rb",
+    "lib/applications/ruby_poi_translations.rb",
+    "lib/applications/spreadsheet_extensions.rb",
     "lib/datashift.rb",
     "lib/datashift/delimiters.rb",
     "lib/datashift/exceptions.rb",
@@ -45,9 +52,9 @@ Gem::Specification.new do |s|
     "lib/generators/csv_generator.rb",
     "lib/generators/excel_generator.rb",
     "lib/generators/generator_base.rb",
+    "lib/guards.rb",
     "lib/helpers/core_ext/to_b.rb",
     "lib/helpers/rake_utils.rb",
-    "lib/helpers/spree_helper.rb",
     "lib/java/poi-3.7/._poi-3.7-20101029.jar5645100390082102460.tmp",
     "lib/java/poi-3.7/LICENSE",
     "lib/java/poi-3.7/NOTICE",
@@ -66,40 +73,22 @@ Gem::Specification.new do |s|
     "lib/loaders/csv_loader.rb",
     "lib/loaders/excel_loader.rb",
     "lib/loaders/loader_base.rb",
-    "lib/loaders/paperclip/image_loader.rb",
-    "lib/loaders/spreadsheet_loader.rb",
-    "lib/loaders/spree/image_loader.rb",
-    "lib/loaders/spree/product_loader.rb",
+    "lib/loaders/paperclip/attachment_loader.rb",
+    "lib/loaders/paperclip/datashift_paperclip.rb",
+    "lib/loaders/paperclip/image_loading.rb",
     "lib/thor/export.thor",
-    "lib/thor/generate_excel.thor",
+    "lib/thor/generate.thor",
     "lib/thor/import.thor",
-    "lib/thor/spree/bootstrap_cleanup.thor",
-    "lib/thor/spree/products_images.thor",
-    "lib/thor/spree/reports.thor",
+    "lib/thor/paperclip.thor",
     "lib/thor/tools.thor",
-    "public/spree/products/large/DEMO_001_ror_bag.jpeg",
-    "public/spree/products/large/DEMO_002_Powerstation.jpg",
-    "public/spree/products/large/DEMO_003_ror_mug.jpeg",
-    "public/spree/products/mini/DEMO_001_ror_bag.jpeg",
-    "public/spree/products/mini/DEMO_002_Powerstation.jpg",
-    "public/spree/products/mini/DEMO_003_ror_mug.jpeg",
-    "public/spree/products/original/DEMO_001_ror_bag.jpeg",
-    "public/spree/products/original/DEMO_002_Powerstation.jpg",
-    "public/spree/products/original/DEMO_003_ror_mug.jpeg",
-    "public/spree/products/product/DEMO_001_ror_bag.jpeg",
-    "public/spree/products/product/DEMO_002_Powerstation.jpg",
-    "public/spree/products/product/DEMO_003_ror_mug.jpeg",
-    "public/spree/products/small/DEMO_001_ror_bag.jpeg",
-    "public/spree/products/small/DEMO_002_Powerstation.jpg",
-    "public/spree/products/small/DEMO_003_ror_mug.jpeg",
     "spec/Gemfile",
     "spec/csv_exporter_spec.rb",
     "spec/csv_loader_spec.rb",
     "spec/datashift_spec.rb",
-    "spec/db/migrate/20110803201325_create_test_bed.rb",
     "spec/excel_exporter_spec.rb",
     "spec/excel_generator_spec.rb",
     "spec/excel_loader_spec.rb",
+    "spec/excel_spec.rb",
     "spec/file_definitions.rb",
     "spec/fixtures/BadAssociationName.xls",
     "spec/fixtures/DemoNegativeTesting.xls",
@@ -109,59 +98,100 @@ Gem::Specification.new do |s|
     "spec/fixtures/ProjectsSingleCategories.xls",
     "spec/fixtures/SimpleProjects.xls",
     "spec/fixtures/config/database.yml",
-    "spec/fixtures/datashift_Spree_db.sqlite",
-    "spec/fixtures/datashift_test_models_db.sqlite",
+    "spec/fixtures/db/datashift_test_models_db.sqlite",
+    "spec/fixtures/db/migrate/20110803201325_create_test_bed.rb",
     "spec/fixtures/images/DEMO_001_ror_bag.jpeg",
     "spec/fixtures/images/DEMO_002_Powerstation.jpg",
     "spec/fixtures/images/DEMO_003_ror_mug.jpeg",
     "spec/fixtures/images/DEMO_004_ror_ringer.jpeg",
-    "spec/fixtures/negative/SpreeProdMiss1Mandatory.csv",
-    "spec/fixtures/negative/SpreeProdMiss1Mandatory.xls",
-    "spec/fixtures/negative/SpreeProdMissManyMandatory.csv",
-    "spec/fixtures/negative/SpreeProdMissManyMandatory.xls",
+    "spec/fixtures/load_datashift.thor",
+    "spec/fixtures/models/category.rb",
+    "spec/fixtures/models/empty.rb",
+    "spec/fixtures/models/loader_release.rb",
+    "spec/fixtures/models/long_and_complex_table_linked_to_version.rb",
+    "spec/fixtures/models/milestone.rb",
+    "spec/fixtures/models/owner.rb",
+    "spec/fixtures/models/project.rb",
+    "spec/fixtures/models/test_model_defs.rb",
+    "spec/fixtures/models/version.rb",
     "spec/fixtures/simple_export_spec.xls",
     "spec/fixtures/simple_template_spec.xls",
-    "spec/fixtures/spree/SpreeImages.xls",
-    "spec/fixtures/spree/SpreeMultiVariant.csv",
-    "spec/fixtures/spree/SpreeProducts.csv",
-    "spec/fixtures/spree/SpreeProducts.xls",
-    "spec/fixtures/spree/SpreeProductsDefaults.yml",
-    "spec/fixtures/spree/SpreeProductsMandatoryOnly.xls",
-    "spec/fixtures/spree/SpreeProductsMultiColumn.csv",
-    "spec/fixtures/spree/SpreeProductsMultiColumn.xls",
-    "spec/fixtures/spree/SpreeProductsSimple.csv",
-    "spec/fixtures/spree/SpreeProductsSimple.xls",
-    "spec/fixtures/spree/SpreeProductsWithImages.csv",
-    "spec/fixtures/spree/SpreeProductsWithImages.xls",
-    "spec/fixtures/spree/SpreeZoneExample.csv",
-    "spec/fixtures/spree/SpreeZoneExample.xls",
     "spec/fixtures/test_model_defs.rb",
     "spec/loader_spec.rb",
     "spec/method_dictionary_spec.rb",
     "spec/method_mapper_spec.rb",
+    "spec/rails_sandbox/.gitignore",
+    "spec/rails_sandbox/Gemfile",
+    "spec/rails_sandbox/README.rdoc",
+    "spec/rails_sandbox/Rakefile",
+    "spec/rails_sandbox/app/assets/images/rails.png",
+    "spec/rails_sandbox/app/assets/javascripts/application.js",
+    "spec/rails_sandbox/app/assets/stylesheets/application.css",
+    "spec/rails_sandbox/app/controllers/application_controller.rb",
+    "spec/rails_sandbox/app/helpers/application_helper.rb",
+    "spec/rails_sandbox/app/mailers/.gitkeep",
+    "spec/rails_sandbox/app/models/.gitkeep",
+    "spec/rails_sandbox/app/models/category.rb",
+    "spec/rails_sandbox/app/models/empty.rb",
+    "spec/rails_sandbox/app/models/loader_release.rb",
+    "spec/rails_sandbox/app/models/long_and_complex_table_linked_to_version.rb",
+    "spec/rails_sandbox/app/models/milestone.rb",
+    "spec/rails_sandbox/app/models/owner.rb",
+    "spec/rails_sandbox/app/models/project.rb",
+    "spec/rails_sandbox/app/models/test_model_defs.rb",
+    "spec/rails_sandbox/app/models/version.rb",
+    "spec/rails_sandbox/app/views/layouts/application.html.erb",
+    "spec/rails_sandbox/config.ru",
+    "spec/rails_sandbox/config/application.rb",
+    "spec/rails_sandbox/config/boot.rb",
+    "spec/rails_sandbox/config/database.yml",
+    "spec/rails_sandbox/config/environment.rb",
+    "spec/rails_sandbox/config/environments/development.rb",
+    "spec/rails_sandbox/config/environments/production.rb",
+    "spec/rails_sandbox/config/environments/test.rb",
+    "spec/rails_sandbox/config/initializers/backtrace_silencers.rb",
+    "spec/rails_sandbox/config/initializers/inflections.rb",
+    "spec/rails_sandbox/config/initializers/mime_types.rb",
+    "spec/rails_sandbox/config/initializers/secret_token.rb",
+    "spec/rails_sandbox/config/initializers/session_store.rb",
+    "spec/rails_sandbox/config/initializers/wrap_parameters.rb",
+    "spec/rails_sandbox/config/locales/en.yml",
+    "spec/rails_sandbox/config/routes.rb",
+    "spec/rails_sandbox/db/migrate/20110803201325_create_test_bed.rb",
+    "spec/rails_sandbox/db/schema.rb",
+    "spec/rails_sandbox/db/seeds.rb",
+    "spec/rails_sandbox/lib/assets/.gitkeep",
+    "spec/rails_sandbox/lib/tasks/.gitkeep",
+    "spec/rails_sandbox/log/.gitkeep",
+    "spec/rails_sandbox/public/404.html",
+    "spec/rails_sandbox/public/422.html",
+    "spec/rails_sandbox/public/500.html",
+    "spec/rails_sandbox/public/favicon.ico",
+    "spec/rails_sandbox/public/index.html",
+    "spec/rails_sandbox/public/robots.txt",
+    "spec/rails_sandbox/script/rails",
+    "spec/rails_sandbox/test/fixtures/.gitkeep",
+    "spec/rails_sandbox/test/functional/.gitkeep",
+    "spec/rails_sandbox/test/integration/.gitkeep",
+    "spec/rails_sandbox/test/performance/browsing_test.rb",
+    "spec/rails_sandbox/test/test_helper.rb",
+    "spec/rails_sandbox/test/unit/.gitkeep",
+    "spec/rails_sandbox/vendor/assets/javascripts/.gitkeep",
+    "spec/rails_sandbox/vendor/assets/stylesheets/.gitkeep",
+    "spec/rails_sandbox/vendor/plugins/.gitkeep",
     "spec/spec_helper.rb",
-    "spec/spree_exporter_spec.rb",
-    "spec/spree_generator_spec.rb",
-    "spec/spree_images_loader_spec.rb",
-    "spec/spree_loader_spec.rb",
-    "spec/spree_method_mapping_spec.rb",
-    "spec/spree_variants_loader_spec.rb",
     "spec/thor_spec.rb",
     "tasks/config/seed_fu_product_template.erb",
     "tasks/config/tidy_config.txt",
     "tasks/db_tasks.rake",
-    "tasks/export/excel_generator.rake",
     "tasks/file_tasks.rake",
-    "tasks/import/excel.rake",
-    "tasks/word_to_seedfu.rake",
-    "test/helper.rb",
-    "test/test_interact.rb"
+    "tasks/word_to_seedfu.rake"
   ]
   s.homepage = "http://github.com/autotelik/datashift"
   s.licenses = ["MIT"]
   s.require_paths = ["lib"]
-  s.rubygems_version = "1.8.24"
-  s.summary = "Shift data betwen applications and Active Record"
+  s.rubygems_version = "1.8.15"
+  s.summary = "Shift data betwen Excel/CSV and any Ruby app"
 
   if s.respond_to? :specification_version then
     s.specification_version = 3