activewarehouse-etl 0.9.1 → 0.9.5.rc1

data/.gitignore

@@ -0,0 +1,7 @@
+pkg/*
+source_data
+test/output/*
+rdoc
+.rvmrc
+.bundle
+*.gem

data/0.9-UPGRADE

@@ -0,0 +1,6 @@
+The 0.9 revision of ActiveWarehouse ETL significantly changes how connections are maintained. This release is not backwards compatible.
+
+To upgrade, you must do the following:
+
+1.) All database connections used in ETL control files must be declared in database.yml in the directory that contains your ETL control files.
+2.) All sources, destinations, transforms and processors that use a database connection must include the configuration name/value pair of :target => 'name' where name is replaced with the connection name defined in database.yml. Connection information should no longer be included in control files.

data/CHANGELOG

@@ -1,89 +1,113 @@
-0.1.0 - Dec 6, 2006
-* Initial release
+0.9.5 - unreleased
+* Fixes for Rails 3 (gkfabs)
+* Fixes for Ruby 1.9.2 (jlecour, byrnejb, thbar)
+* improvements on CalculationTransform (gkfabs)
+* batch can have children (gkfabs)
+* read all columns by default in DelimitedSource (smeyfroi)
+* Turn db config into a HashWithIndifferentAccess (smeyfroi)
+* DatabaseJoinProcessor (gkfabs)
+* ImapattachmentDownloaderProcessor (gkfabs)
+* Pop3attachementDownloaderProcessor (gkfabs)
+* Enhancements and fixes on EscapeCSVProcessor (gkfabs)
+* Bug correction on hash initialization DatabaseSource (gkfabs)
+* FilterRowProcessor won't return nil (gkfabs)
+* DatabaseSource fixes (gkfabs)
+* New UpdateDatabaseDestination (gkfabs)
+* New InsertUpdateDatabaseDestination (gkfabs)
+* New Excel destination (gkfabs)
+* New Excel parser (gkfabs)
+* Add ability to use a query in DatabaseSource (gkfabs)
+* Modified SQLResolver to allow for multiple key lookups on dimension
+table (cdimartino)
+* Add scd_required_fields which allow to avoid specifying the scd fields when specifying :unique in a destination (darrell)
+* More SCD debug logging (darrell)
+* Make database destination table quoting database agnostic (darrell)
+* Allow absolute paths in file destinations and file bulk imports (darrell)
+* Bug fixes on engine, destination (sasikumargn, cdimartino, darrell)
+* Code and tests clean-up (mainej, aeden, thbar)
+* No more failure on tests (thbar, gkfabs)
+* Add the ability to automatically run the tests on a matrix of configuration including Rails 2/3, Ruby 1.8.7/1.9.2, MySQL/Postgresql (thbar)
+* Use bundler for gem packaging and release (thbar)
+
+0.9.1 - January 14, 2009
+* SQLResolver now uses ETL::Engine.table so it may utilize temp tables. (aeden)
+* Added Thibaut Barrère's encode processor.
+* Added MockSource and MockDestination test helpers (thbar)
+* Added the block processor. Can call a block once (pre/post processor)
+  or once for each row (after_read/before_write row processor) (thbar)
+* Changed temp table to use new AdapterExtension copy_table method (aeden)
+* Added bin/etl.cmd windows batch - just add the bin folder to your PATH
+  and it will let you call etl on an unpacked/pistoned version of AW-ETL (thbar)
+* Upgraded to support Rails 2.1. No longer compatible with older versions of Rails.
+* Added ETL::Builder::TimeDimensionBuilder
+* Added :default option to ForeignKeyLookupTransform that will be used if no
+  foreign key is found.
+* Added :cache option to ForeignKeyLookupTransform that will preload the FK
+  mappings if the underlying resolver supports it. Currently supported by 
+  SQLResolver.
+* A Class extending ETL::Transform::Transform may now be passed as a transformer.
+  For example, in the control file you would define the transform as:
+  transform :a_field, MyTransform, {:option1 => 'option1'}.
+* Support Ruby 1.9 CSV library
 
-0.2.0 - Dec 7, 2006
-* Added an XML parser for source parsing
-* Added support for compound key constraints in destinations via the 
-  :unique => [] option
-* Added ability to declare explicit columns in bulk import
-* Added support for generators in destinations
-* Added a SurrogateKeyGenerator for cases where the database doesn't support
-  auto generated surrogate keys
+0.9.0 - August 9, 2007
+* Added support for batch processing through .ebf files. These files are
+  essentially control files that apply settings to an entire ETL process.
+* Implemented support for screen blocks. These blocks can be used to test
+  the data and raise an error if the screens do not pass.
+* Connections are now cached in a Hash available through 
+  ETL::Engine.connection(name). This should be used rather than including
+  connection information in the control files.
+* Implemented temp table support throughout.
+* DateDimensionBuilder now included in ActiveWarehouse ETL directly.
+* Time calculations for fiscal year now included in ActiveWarehouse ETL.
 
-0.3.0 - Dec 19, 2006
-* Added support for calculated values in virtual fields with Proc
+0.8.4 - May 24, 2007
+* Added fix for backslash in file writer
 
-0.4.0 - Jan 11, 2006
-* Added :skip_lines option to file source configurations, which can be used
-  to skip the first n lines in the source data file
-* Added better error handling in delimited parser - an error is now raised 
-  if the expected and actual field lengths do not match
-* Added :truncate option for database destination. Set to true to truncate 
-  before importing data.
-* Added support for :unique => [] option and virtual fields for the database 
-  destination
+0.8.3 - May 13, 2007
+* Added patches from Andy Triboletti
 
-0.5.0 - Feb 17, 2007
-* Changed require_gem to gem and added alias to allow for older versions of 
-  rubygems.
-* Added support for Hash in the source configuration where :name => :parser_name
-  defines the parser to use and :options => {} defines options to pass to the 
-  parser.
-* Added support for passing a custom Parser class in the source configuration.
-* Removed the need to include Enumerable in each parser implementation.
-* Added new date_to_string and string_to_date transformers.
-* Implemented foreign_key_lookup transform including an ActiveRecordResolver.
-* Added real time activity logging which is called when the etl bin script is 
-  invoked.
-* Improved error handling.
-* Default logger level is now WARN.
+0.8.2 - April 15, 2007
+* Fixed bug with premature destination closing.
+* Added indexes to execution records table.
+* Added a PrintRowProcessor.
+* Added support for conditions and "group by" in the database source.
+* Added after_initialize hook in Processor base class.
+* Added examples directory
 
-0.5.1 - Feb 18, 2007
-* Fixed up truncate processor.
-* Updated HOW_TO_RELEASE doc.
+0.8.1 - Apr 12, 2007
+* Added EnumerableSource
+* Added :type configuration option to the source directive, allowing the source
+  type to be explicitly specified. The source type can be a string or symbol 
+  (in which case the class will be constructed by appending Source to the type 
+  name), a class (which will be instantiate and passed the control, 
+  configuration and mapping) and finally an actual Source instance.
 
-0.5.2 - Feb 19, 2007
-* Added error threshold.
-* Fixed problem with transform error handling.
+0.8.0 - Apr 12, 2007
+* Source now available through the current row source accessor.
+* Added new_rows_only configuration option to DatabaseSource. A date field must
+  be specified and only records that are greater than the date value in that 
+  field, relative to the last successful
+  execution, will be returned from the source.
+* Added an (untested) count feature which returns the number of rows for 
+  processing.
+* If no natural key is defined then an empty array will now be used, resulting
+  in the row being written to the output without going through change checks.
+* Mapping argument in destination is now optional. An empty hash will be used
+  if the mapping hash is not specified. If the mapping hash is not specified 
+  then the order will be determined using the originating source's order.
+* ActiveRecord configurations loaded from database.yml by the etl tool will be 
+  merged with ActiveRecord::Base.configurations.
+* Fixed several bugs in how record change detection was implemented.
+* Fixed how the read_locally functionality was implemented so that it will find
+  that last completed local source copy using the source's trigger file (untested).
 
-0.6.0 - Mar 8, 2007
-* Fixed missing method problem in validate in Control class.
-* Removed control validation for now (source could be code in the control file).
-* Transform interface now defined as taking 3 arguments, the field name, field 
-  value and the row. This is not backwards compatible.
-* Added HierarchyLookupTransform.
-* Added DefaultTransform which will return a specified value if the initial 
-  value is blank.
-* Added row-level processing.
-* Added HierarchyExploderProcessor which takes a single hierarchy row and 
-  explodes it to multiple rows as used in a hierarchy bridge.
-* Added ApacheCombinedLogParser which parses Apache Combined Log format, 
-  including parsing of the 
-  user agent string and the URI, returning a Hash.
-* Fixed bug in SAX parser so that attributes are now set when the start_element
-  event is received.
-* Added an HttpTools module which provides some parsing methods (for user agent
-  and URI).
-* Database source now uses its own class for establishing an ActiveRecord 
-  connection.
-* Log files are now timestamped.
-* Source files are now archived automatically during the extraction process
-* Added a :condition option to the destination configuration Hash that accepts 
-  a Proc with a single argument passed to it (the row).
-* Added an :append_rows option to the destination configuration Hash that 
-  accepts either a Hash (to append a single row) or an Array of Hashes (to 
-  append multiple rows).
-* Only print the read and written row counts if there is at least one source 
-  and one destination respectively.
-* Added a depends_on directive that accepts a list of arguments of either strings 
-  or symbols. Each symbol is converted to a string and .ctl is appended; 
-  strings are passed through directly. The dependencies are executed in the order 
-  they are specified.
-* The default field separator in the bulk loader is now a comma (was a tab).
+0.7.2 - Apr 8, 2007
+* Fixed quoting bug in CheckExistProcessor
 
-0.6.1 - Mar 22, 2007
-* Added support for absolute paths in file sources
-* Added CopyFieldProcessor
+0.7.1 - Apr 8, 2007
+* Fixed source caching
 
 0.7 - Apr 8, 2007
 * Job execution is now tracked in a database. This means that ActiveRecord is 
@@ -118,81 +142,89 @@
   This is useful if you need to recover from a failed ETL process.
 * Updated README
 
-0.7.1 - Apr 8, 2007
-* Fixed source caching
+0.6.1 - Mar 22, 2007
+* Added support for absolute paths in file sources
+* Added CopyFieldProcessor
 
-0.7.2 - Apr 8, 2007
-* Fixed quoting bug in CheckExistProcessor
+0.6.0 - Mar 8, 2007
+* Fixed missing method problem in validate in Control class.
+* Removed control validation for now (source could be code in the control file).
+* Transform interface now defined as taking 3 arguments, the field name, field 
+  value and the row. This is not backwards compatible.
+* Added HierarchyLookupTransform.
+* Added DefaultTransform which will return a specified value if the initial 
+  value is blank.
+* Added row-level processing.
+* Added HierarchyExploderProcessor which takes a single hierarchy row and 
+  explodes it to multiple rows as used in a hierarchy bridge.
+* Added ApacheCombinedLogParser which parses Apache Combined Log format, 
+  including parsing of the 
+  user agent string and the URI, returning a Hash.
+* Fixed bug in SAX parser so that attributes are now set when the start_element
+  event is received.
+* Added an HttpTools module which provides some parsing methods (for user agent
+  and URI).
+* Database source now uses its own class for establishing an ActiveRecord 
+  connection.
+* Log files are now timestamped.
+* Source files are now archived automatically during the extraction process
+* Added a :condition option to the destination configuration Hash that accepts 
+  a Proc with a single argument passed to it (the row).
+* Added an :append_rows option to the destination configuration Hash that 
+  accepts either a Hash (to append a single row) or an Array of Hashes (to 
+  append multiple rows).
+* Only print the read and written row counts if there is at least one source 
+  and one destination respectively.
+* Added a depends_on directive that accepts a list of arguments of either strings 
+  or symbols. Each symbol is converted to a string and .ctl is appended; 
+  strings are passed through directly. The dependencies are executed in the order 
+  they are specified.
+* The default field separator in the bulk loader is now a comma (was a tab).
 
-0.8.0 - Apr 12, 2007
-* Source now available through the current row source accessor.
-* Added new_rows_only configuration option to DatabaseSource. A date field must
-  be specified and only records that are greater than the date value in that 
-  field, relative to the last successful
-  execution, will be returned from the source.
-* Added an (untested) count feature which returns the number of rows for 
-  processing.
-* If no natural key is defined then an empty array will now be used, resulting
-  in the row being written to the output without going through change checks.
-* Mapping argument in destination is now optional. An empty hash will be used
-  if the mapping hash is not specified. If the mapping hash is not specified 
-  then the order will be determined using the originating source's order.
-* ActiveRecord configurations loaded from database.yml by the etl tool will be 
-  merged with ActiveRecord::Base.configurations.
-* Fixed several bugs in how record change detection was implemented.
-* Fixed how the read_locally functionality was implemented so that it will find
-  that last completed local source copy using the source's trigger file (untested).
-  
-0.8.1 - Apr 12, 2007
-* Added EnumerableSource
-* Added :type configuration option to the source directive, allowing the source
-  type to be explicitly specified. The source type can be a string or symbol 
-  (in which case the class will be constructed by appending Source to the type 
-  name), a class (which will be instantiate and passed the control, 
-  configuration and mapping) and finally an actual Source instance.
-  
-0.8.2 - April 15, 2007
-* Fixed bug with premature destination closing.
-* Added indexes to execution records table.
-* Added a PrintRowProcessor.
-* Added support for conditions and "group by" in the database source.
-* Added after_initialize hook in Processor base class.
-* Added examples directory
+0.5.2 - Feb 19, 2007
+* Added error threshold.
+* Fixed problem with transform error handling.
 
-0.8.3 - May 13, 2007
-* Added patches from Andy Triboletti
+0.5.1 - Feb 18, 2007
+* Fixed up truncate processor.
+* Updated HOW_TO_RELEASE doc.
 
-0.8.4 - May 24, 2007
-* Added fix for backslash in file writer
+0.5.0 - Feb 17, 2007
+* Changed require_gem to gem and added alias to allow for older versions of 
+  rubygems.
+* Added support for Hash in the source configuration where :name => :parser_name
+  defines the parser to use and :options => {} defines options to pass to the 
+  parser.
+* Added support for passing a custom Parser class in the source configuration.
+* Removed the need to include Enumerable in each parser implementation.
+* Added new date_to_string and string_to_date transformers.
+* Implemented foreign_key_lookup transform including an ActiveRecordResolver.
+* Added real time activity logging which is called when the etl bin script is 
+  invoked.
+* Improved error handling.
+* Default logger level is now WARN.
 
-0.9.0 - August 9, 2007
-* Added support for batch processing through .ebf files. These files are
-  essentially control files that apply settings to an entire ETL process.
-* Implemented support for screen blocks. These blocks can be used to test
-  the data and raise an error if the screens do not pass.
-* Connections are now cached in a Hash available through 
-  ETL::Engine.connection(name). This should be used rather than including
-  connection information in the control files.
-* Implemented temp table support throughout.
-* DateDimensionBuilder now included in ActiveWarehouse ETL directly.
-* Time calculations for fiscal year now included in ActiveWarehouse ETL.
+0.4.0 - Jan 11, 2006
+* Added :skip_lines option to file source configurations, which can be used
+  to skip the first n lines in the source data file
+* Added better error handling in delimited parser - an error is now raised 
+  if the expected and actual field lengths do not match
+* Added :truncate option for database destination. Set to true to truncate 
+  before importing data.
+* Added support for :unique => [] option and virtual fields for the database 
+  destination
 
-0.9.1 -
-* SQLResolver now uses ETL::Engine.table so it may utilize temp tables. (aeden)
-* Added Thibaut Barrère's encode processor.
-* Added MockSource and MockDestination test helpers (thbar)
-* Added the block processor. Can call a block once (pre/post processor)
-  or once for each row (after_read/before_write row processor) (thbar)
-* Changed temp table to use new AdapterExtension copy_table method (aeden)
-* Added bin/etl.cmd windows batch - just add the bin folder to your PATH
-  and it will let you call etl on an unpacked/pistoned version of AW-ETL (thbar)
-* Upgraded to support Rails 2.1. No longer compatible with older versions of Rails.
-* Added ETL::Builder::TimeDimensionBuilder
-* Added :default option to ForeignKeyLookupTransform that will be used if no
-  foreign key is found.
-* Added :cache option to ForeignKeyLookupTransform that will preload the FK
-  mappings if the underlying resolver supports it. Currently supported by 
-  SQLResolver.
-* A Class extending ETL::Transform::Transform may now be passed as a transformer.
-  For example, in the control file you would define the transform as:
-  transform :a_field, MyTransform, {:option1 => 'option1'}.
+0.3.0 - Dec 19, 2006
+* Added support for calculated values in virtual fields with Proc
+
+0.2.0 - Dec 7, 2006
+* Added an XML parser for source parsing
+* Added support for compound key constraints in destinations via the 
+  :unique => [] option
+* Added ability to declare explicit columns in bulk import
+* Added support for generators in destinations
+* Added a SurrogateKeyGenerator for cases where the database doesn't support
+  auto generated surrogate keys
+
+0.1.0 - Dec 6, 2006
+* Initial release

data/Gemfile

@@ -0,0 +1,4 @@
+source "http://rubygems.org"
+
+# Specify your gem's dependencies in ..gemspec
+gemspec

data/HOW_TO_RELEASE

@@ -0,0 +1,9 @@
+* update lib/etl/version
+* push your changes
+* then use bundler to build + git tag + push to rubygems
+
+    rake release
+
+* you can list changes using github:
+
+    https://github.com/activewarehouse/activewarehouse-etl/compare/release-0.9.1...master

data/README

@@ -7,7 +7,7 @@ Ruby Extract-Transform-Load (ETL) tool.
 
 == Online Documentation
 
-Available at http://activewarehouse.rubyforge.org/docs/activewarehouse-etl.html
+Available at https://github.com/activewarehouse/activewarehouse-etl/wiki
 
 == Features
 
@@ -77,7 +77,23 @@ Command line options:
 Control file examples can be found in the examples directory.
 
 == Running Tests
-The tests require Shoulda 1.x.
+
+Current state:
+- 11 failures on MySQL
+- 1 failure on Postgres
+
+The tests require:
+- gem install shoulda
+- gem install flexmock
+- gem install pg (if you want to run the tests on pg)
+- gem install spreadsheet
+- gem install tmail
+
+The tests subfolder contains examples database.yml for mysql and postgres.
+
+To run the tests:
+- rake test DB=postgresql (for postgres)
+- otherwise just rake test
 
 == Feedback
 This is a work in progress. Comments should be made on the 

data/Rakefile

@@ -1,21 +1,41 @@
+require 'bundler/gem_tasks'
+
 require 'rake'
 require 'rake/testtask'
-require 'rake/rdoctask'
-require 'rake/packagetask'
-require 'rake/gempackagetask'
-require 'rake/contrib/rubyforgepublisher'
-
-require File.join(File.dirname(__FILE__), 'lib/etl', 'version')
-
-module AWETL
-  PKG_BUILD       = ENV['PKG_BUILD'] ? '.' + ENV['PKG_BUILD'] : ''
-  PKG_NAME        = 'activewarehouse-etl'
-  PKG_VERSION     = ETL::VERSION::STRING + PKG_BUILD
-  PKG_FILE_NAME   = "#{PKG_NAME}-#{PKG_VERSION}"
-  PKG_DESTINATION = ENV["PKG_DESTINATION"] || "../#{PKG_NAME}"
+require 'rdoc'
+require 'rdoc/task'
+
+namespace :test do
+
+  def run_tests(rvm, rails, database)
+    database_yml = File.dirname(__FILE__) + "/test/config/database.#{database}.yml"
+    FileUtils.cp(database_yml, 'test/database.yml')
+
+    puts
+    puts "============ Ruby #{rvm} - Rails #{rails} - Db #{database} ============="
+    puts
+
+    sh <<-BASH
+    BUNDLE_GEMFILE=test/config/Gemfile.rails-#{rails} bundle install > null
+    BUNDLE_GEMFILE=test/config/Gemfile.rails-#{rails} rvm #{rvm} rake test
+  BASH
+  end
+
+  desc 'Run the tests in all combinations described in test-matrix.yml'
+  task :matrix do
+    # a la travis
+    require 'yaml'
+    data = YAML.load(IO.read(File.dirname(__FILE__) + '/test-matrix.yml'))
+    data['rvm'].each do |rvm|
+      data['rails'].each do |rails|
+        data['database'].each do |database|
+          run_tests(rvm, rails, database)
+        end
+      end
+    end
+  end
 end
 
-desc 'Default: run unit tests.'
 task :default => :test
 
 desc 'Test the ETL application.'
@@ -46,62 +66,6 @@ namespace :rcov do
   end
 end
 
-# Gem Spec
-
-module AWETL
-  def self.package_files(package_prefix)
-    FileList[
-      "#{package_prefix}CHANGELOG",
-      "#{package_prefix}LICENSE",
-      "#{package_prefix}README",
-      "#{package_prefix}TODO",
-      "#{package_prefix}Rakefile",
-      "#{package_prefix}bin/**/*",
-      "#{package_prefix}doc/**/*",
-      "#{package_prefix}lib/**/*",
-      "#{package_prefix}examples/**/*",
-    ] - [ "#{package_prefix}test" ]
-  end
-
-  def self.spec(package_prefix = '')
-    Gem::Specification.new do |s|
-      s.name = 'activewarehouse-etl'
-      s.version = AWETL::PKG_VERSION
-      s.summary = "Pure Ruby ETL package."
-      s.description = <<-EOF
-        ActiveWarehouse ETL is a pure Ruby Extract-Transform-Load application for loading data into a database.
-      EOF
-
-      s.add_dependency('rake',                '>= 0.7.1')
-      s.add_dependency('activesupport',       '>= 1.3.1')
-      s.add_dependency('activerecord',        '>= 1.14.4')
-      s.add_dependency('fastercsv',           '>= 1.2.0')
-      s.add_dependency('adapter_extensions',  '>= 0.1.0')
-
-      s.rdoc_options << '--exclude' << '.'
-      s.has_rdoc = false
-
-      s.files = package_files(package_prefix).to_a.delete_if {|f| f.include?('.svn')}
-      s.require_path = 'lib'
-
-      s.bindir = "#{package_prefix}bin" # Use these for applications.
-      s.executables = ['etl']
-      s.default_executable = "etl"
-
-      s.author = "Anthony Eden"
-      s.email = "anthonyeden@gmail.com"
-      s.homepage = "http://activewarehouse.rubyforge.org/etl"
-      s.rubyforge_project = "activewarehouse"
-    end
-  end
-end
-
-Rake::GemPackageTask.new(AWETL.spec) do |pkg|
-  pkg.gem_spec = AWETL.spec
-  pkg.need_tar = true
-  pkg.need_zip = true
-end
-
 desc "Generate code statistics"
 task :lines do
   lines, codelines, total_lines, total_codelines = 0, 0, 0, 0
@@ -127,27 +91,7 @@ task :lines do
   puts "Total: Lines #{total_lines}, LOC #{total_codelines}"
 end
 
-desc "Publish the release files to RubyForge."
-task :release => [ :package ] do
-  `rubyforge login`
-
-  for ext in %w( gem tgz zip )
-    release_command = "rubyforge add_release activewarehouse #{AWETL::PKG_NAME} 'REL #{AWETL::PKG_VERSION}' pkg/#{AWETL::PKG_NAME}-#{AWETL::PKG_VERSION}.#{ext}"
-    puts release_command
-    system(release_command)
-  end
-end
-
-desc "Publish the API documentation"
+desc "Publish the API documentation (UNTESTED CURRENTLY)"
 task :pdoc => [:rdoc] do 
   Rake::SshDirPublisher.new("aeden@rubyforge.org", "/var/www/gforge-projects/activewarehouse/etl/rdoc", "rdoc").upload
 end
-
-desc "Reinstall the gem from a local package copy"
-task :reinstall => [:package] do
-  windows = RUBY_PLATFORM =~ /mswin/
-  sudo = windows ? '' : 'sudo'
-  gem = windows ? 'gem.bat' : 'gem'
-  `#{sudo} #{gem} uninstall #{AWETL::PKG_NAME} -x`
-  `#{sudo} #{gem} install pkg/#{AWETL::PKG_NAME}-#{AWETL::PKG_VERSION}`
-end