activewarehouse-etl 0.9.5.rc1 → 1.0.0.rc1

data/.standalone_migrations

@@ -0,0 +1,2 @@
+config:
+  database: test/config/database.yml

data/.travis.yml

@@ -0,0 +1,15 @@
+language: ruby
+gemfile:
+  - test/config/gemfiles/Gemfile.rails-3.2.x
+  - test/config/gemfiles/Gemfile.rails-3.1.x
+  - test/config/gemfiles/Gemfile.rails-3.0.x
+rvm:
+  - 1.9.3
+  - 1.8.7
+env:
+  - DB=mysql2
+  - DB=postgresql
+before_script:
+  - bundle exec rake db:create RAILS_ENV=$DB
+  - bundle exec rake db:create RAILS_ENV=etl_execution
+  - bundle exec rake db:schema:load RAILS_ENV=$DB

data/CHANGELOG

@@ -1,9 +1,18 @@
 0.9.5 - unreleased
+* EnsureFieldsPresenceProcessor treat symbols and strings equally (thbar)
+* BREAKING CHANGE: you must require 'iconv' if you use the :encode_processor from now on (thbar)
+* A non-zero exit code should be returned on fatal screen or exceeded error threshold (thbar)
+* BREAKING CHANGE: the TruncateProcessor will now RESTART IDENTITY when using postgres - this behaviour can be changed by passing :options => 'CONTINUE IDENTITY' (thbar)
+* New EnsureFieldsPresenceProcessor (thbar)
+* New CsvDestination based on FasterCSV (lgustafson)
+* Fix #41: Allow cache to be turned off on ForeignKeyLookupTransform
+* Fix #36: etl --read-locally picks wrong last file (lgustafson)
+* BREAKING CHANGE: DelimitedParser has been renamed to CsvParser, to emphasize the fact that this parser relies on FasterCSV under the hood (which means it will choke on faulty data).
 * Fixes for Rails 3 (gkfabs)
 * Fixes for Ruby 1.9.2 (jlecour, byrnejb, thbar)
 * improvements on CalculationTransform (gkfabs)
 * batch can have children (gkfabs)
-* read all columns by default in DelimitedSource (smeyfroi)
+* read all columns by default in CsvParser (smeyfroi)
 * Turn db config into a HashWithIndifferentAccess (smeyfroi)
 * DatabaseJoinProcessor (gkfabs)
 * ImapattachmentDownloaderProcessor (gkfabs)

data/HOW_TO_RELEASE

@@ -4,6 +4,10 @@
 
     rake release
 
+* if you remain stuck at "Pushed git commits and tags", the task may silently wait for your password. Check this if it's the case:
+
+    https://github.com/carlhuda/bundler/issues/980
+
 * you can list changes using github:
 
     https://github.com/activewarehouse/activewarehouse-etl/compare/release-0.9.1...master

data/LICENSE

@@ -1,4 +1,4 @@
-Copyright (c) 2006-2007 Anthony Eden
+Copyright (c) 2006-2011 Anthony Eden, Thibaut Barrère
 
 Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
 

data/README.textile

@@ -0,0 +1,111 @@
+h1. ActiveWarehouse-ETL
+
+ActiveWarehouse-ETL is a Ruby Extract-Transform-Load (ETL) tool.
+
+This tool is both usable and used in production under its current form - but be aware the project is under reorganization: a new team is shaping up and we're working mostly on making it easier for people to contribute first. Up-to-date documentation will only come later.
+
+"!https://secure.travis-ci.org/activewarehouse/activewarehouse-etl.png!":http://travis-ci.org/activewarehouse/activewarehouse-etl
+
+h2. Usage
+
+The documentation is sparse and not everything is up to date, too, but here are useful bits to get you started:
+
+* read the "Introduction":https://github.com/activewarehouse/activewarehouse-etl/wiki/Documentation
+* later on, refer to the "RDoc":http://rdoc.info/github/activewarehouse/activewarehouse-etl/master/frames (be sure to check out Processor and Transform)
+* read the "source":https://github.com/activewarehouse/activewarehouse-etl/tree/master/lib/etl
+
+If you're lost, please ask questions on the "Google Group":http://groups.google.com/group/activewarehouse-discuss and we'll take care of it.
+
+One thing to keep in mind is that ActiveWarehouse-ETL is highly hackable: you can pretty much create all you need with extra ruby code, even if it's not currently supported.
+
+h2. Compatibility
+
+See "Travis":http://travis-ci.org/activewarehouse/activewarehouse-etl and our set of "gemfiles":https://github.com/activewarehouse/activewarehouse-etl/tree/master/test/config/gemfiles for the currently tested combinations.
+
+If you meet any error, please drop a line on the "Google Group":http://groups.google.com/group/activewarehouse-discuss so that we can fix it.
+
+h2. Contributing
+
+Fork on GitHub and after you've committed tested patches, send a pull request.
+
+TODO: explain how to run the tests like on Travis.
+
+* install RVM and Bundler
+* install MySQL and/or Postgresql (you can use brew for that)
+* edit test/config/database.yml if needed (don't commit this file though)
+* run (for instance):
+
+    @rake "ci:run_one[mysql2,test/config/gemfiles/Gemfile.rails-3.0.x]"@
+
+or for all the combinations:
+    
+    @rake ci:run_matrix@
+
+h2. Contributors
+
+ActiveWarehouse-ETL is the work of many people since late 2006 - here is a list, in no particular order:
+
+* Anthony Eden
+* Chris DiMartino
+* Darrell Fuhriman
+* Fabien Carrion
+* Jacob Maine
+* James B. Byrne
+* Jay Zeschin
+* Jeremy Lecour
+* Steve Meyfroidt
+* Seth Ladd
+* Thibaut Barrère
+* Stephen Touset
+* sasikumargn
+* Andrew Kuklewicz
+* Leif Gustafson
+* Andrew Sodt
+* Tyler Kiley
+* Colman Nady
+* Scott Gonyea
+
+If your name should be on the list but isn't, please leave a comment!
+
+h2. Features
+
+Currently supported features:
+
+* ETL Domain Specific Language (DSL) - Control files are specified in a Ruby-based DSL
+* Multiple source types. Current supported types:
+** Fixed-width and delimited text files
+** XML files through SAX
+** Apache combined log format
+* Multiple destination types - file and database destinations
+* Support for extracting from multiple sources in a single job
+* Support for writing to multiple destinations in a single job
+* A variety of built-in transformations are included:
+** Date-to-string, string-to-date, string-to-datetime, string-to-timestamp
+** Type transformation supporting strings, integers, floats and big decimals
+** Trim
+** SHA-1
+** Decode from an external decode file
+** Default replacement for empty values
+** Ordinalize
+** Hierarchy lookup
+** Foreign key lookup
+** Ruby blocks
+** Any custom transformation class
+* A variety of build-in row-level processors
+** Check exists processor to determine if the record already exists in the destination database
+** Check unique processor to determine whether a matching record was processed during this job execution
+** Copy field
+** Rename field
+** Hierarchy exploder which takes a tree structure defined through a parent id and explodes it into a hierarchy bridge table
+** Surrogate key generator including support for looking up the last surrogate key from the target table using a custom query
+** Sequence generator including support for context-sensitive sequences where the context can be defined as a combination of fields from the source data
+** New row-level processors can easily be defined and applied
+* Pre-processing
+** Truncate processor
+* Post-processing
+** Bulk import using native RDBMS bulk loader tools
+* Virtual fields - Add a field to the destination data which doesn't exist in the source data
+* Built in job and record meta data
+* Support for type 1 and type 2 slowly changing dimensions
+** Automated effective date and end date time stamping for type 2
+** CRC checking

data/Rakefile

@@ -1,97 +1,56 @@
-require 'bundler/gem_tasks'
-
+require 'bundler'
+Bundler::GemHelper.install_tasks
 require 'rake'
 require 'rake/testtask'
-require 'rdoc'
-require 'rdoc/task'
-
-namespace :test do
 
-  def run_tests(rvm, rails, database)
-    database_yml = File.dirname(__FILE__) + "/test/config/database.#{database}.yml"
-    FileUtils.cp(database_yml, 'test/database.yml')
+def system!(cmd)
+  puts cmd
+  raise "Command failed!" unless system(cmd)
+end
 
-    puts
-    puts "============ Ruby #{rvm} - Rails #{rails} - Db #{database} ============="
-    puts
+begin
+  require 'tasks/standalone_migrations'
+rescue LoadError => e
+  puts "gem install standalone_migrations to get db:migrate:* tasks! (Error: #{e})"
+end
 
-    sh <<-BASH
-    BUNDLE_GEMFILE=test/config/Gemfile.rails-#{rails} bundle install > null
-    BUNDLE_GEMFILE=test/config/Gemfile.rails-#{rails} rvm #{rvm} rake test
-  BASH
+# experimental tasks to reproduce the Travis behaviour locally
+namespace :ci do
+
+  desc "For current RVM, run the tests for one db and one gemfile"
+  task :run_one, :db, :gemfile do |t, args|
+    ENV['BUNDLE_GEMFILE'] = File.expand_path(args[:gemfile] || (File.dirname(__FILE__) + '/test/config/gemfiles/Gemfile.rails-3.2.x'))
+    ENV['DB'] = args[:db] || 'mysql2'
+    system! "bundle install"
+    system! "bundle exec rake db:create"
+    system! "bundle exec rake db:create RAILS_ENV=etl_execution"
+    system! "bundle exec rake db:schema:load"
+    system! "bundle exec rake"
   end
 
-  desc 'Run the tests in all combinations described in test-matrix.yml'
-  task :matrix do
-    # a la travis
-    require 'yaml'
-    data = YAML.load(IO.read(File.dirname(__FILE__) + '/test-matrix.yml'))
-    data['rvm'].each do |rvm|
-      data['rails'].each do |rails|
-        data['database'].each do |database|
-          run_tests(rvm, rails, database)
-        end
-      end
+  desc "For current RVM, run the tests for all the combination in travis configuration"
+  task :run_matrix do
+    require 'cartesian'
+    config = YAML.load_file('.travis.yml')
+    config['env'].cartesian(config['gemfile']).each do |*x|
+      env, gemfile = *x.flatten
+      db = env.gsub('DB=', '')
+      print [db, gemfile].inspect.ljust(40) + ": "
+      cmd = "rake \"ci:run_one[#{db},#{gemfile}]\""
+      result = system "#{cmd} > /dev/null 2>&1"
+      result = result ? "OK" : "FAILED! - re-run with: #{cmd}"
+      puts result
     end
   end
+
 end
 
 task :default => :test
 
 desc 'Test the ETL application.'
 Rake::TestTask.new(:test) do |t|
-  t.libs << 'lib'
+  t.libs << 'lib' << '.'
   t.pattern = 'test/**/*_test.rb'
   t.verbose = true
   # TODO: reset the database
 end
-
-desc 'Generate documentation for the ETL application.'
-Rake::RDocTask.new(:rdoc) do |rdoc|
-  rdoc.rdoc_dir = 'rdoc'
-  rdoc.title    = 'ActiveWarehouse ETL'
-  rdoc.options << '--line-numbers' << '--inline-source'
-  rdoc.rdoc_files.include('README')
-  rdoc.rdoc_files.include('lib/**/*.rb')
-end
-
-namespace :rcov do
-  desc 'Measures test coverage'
-  task :test do
-    rm_f 'coverage.data'
-    mkdir 'coverage' unless File.exist?('coverage')
-    rcov = "rcov --aggregate coverage.data --text-summary -Ilib"
-    system("#{rcov} test/*_test.rb")
-    # system("open coverage/index.html") if PLATFORM['darwin']
-  end
-end
-
-desc "Generate code statistics"
-task :lines do
-  lines, codelines, total_lines, total_codelines = 0, 0, 0, 0
-
-  for file_name in FileList["lib/**/*.rb"]
-    next if file_name =~ /vendor/
-    f = File.open(file_name)
-
-    while line = f.gets
-      lines += 1
-      next if line =~ /^\s*$/
-      next if line =~ /^\s*#/
-      codelines += 1
-    end
-    puts "L: #{sprintf("%4d", lines)}, LOC #{sprintf("%4d", codelines)} | #{file_name}"
-
-    total_lines     += lines
-    total_codelines += codelines
-
-    lines, codelines = 0, 0
-  end
-
-  puts "Total: Lines #{total_lines}, LOC #{total_codelines}"
-end
-
-desc "Publish the API documentation (UNTESTED CURRENTLY)"
-task :pdoc => [:rdoc] do 
-  Rake::SshDirPublisher.new("aeden@rubyforge.org", "/var/www/gforge-projects/activewarehouse/etl/rdoc", "rdoc").upload
-end

data/activewarehouse-etl.gemspec

@@ -17,11 +17,14 @@ Gem::Specification.new do |s|
   s.required_rubygems_version = ">= 1.3.6"
 
   s.add_runtime_dependency('rake',                '>= 0.8.3')
-  s.add_runtime_dependency('activesupport',       '>= 2.1.0')
-  s.add_runtime_dependency('activerecord',        '>= 2.1.0')
+  s.add_runtime_dependency('activesupport',       '>= 3.0.0')
+  s.add_runtime_dependency('activerecord',        '>= 3.0.0')
   s.add_runtime_dependency('fastercsv',           '>= 1.2.0')
-  s.add_runtime_dependency('adapter_extensions',  '>= 0.5.0')
-  s.add_runtime_dependency('spreadsheet')
+  s.add_runtime_dependency('adapter_extensions',  '>= 0.9.5.rc1')
+
+  s.add_development_dependency('shoulda', '~>2.11.3')
+  s.add_development_dependency('flexmock', '~>0.9.0')
+  s.add_development_dependency('cartesian')
 
   s.files              = `git ls-files`.split("\n")
   s.test_files         = `git ls-files -- {test}/*`.split("\n")

data/db/migrate/20120229203554_create_tables.rb

@@ -0,0 +1,37 @@
+class CreateTables < ActiveRecord::Migration
+  def self.up
+    create_table(:people, :force => true) do |t|
+      t.column :first_name, :string
+      t.column :last_name, :string
+      t.column :ssn, :string, :limit => 64
+    end
+
+    create_table(:places, :force => true) do |t|
+      t.column :address, :text
+      t.column :city, :string
+      t.column :state, :string
+      t.column :country, :string, :limit => 2
+    end
+
+    create_table(:person_dimension, :force => true) do |t|
+      t.column :first_name, :string, :limit => 50
+      t.column :last_name, :string, :limit => 50
+      t.column :address, :string, :limit => 100
+      t.column :city, :string, :limit => 50
+      t.column :state, :string, :limit => 50
+      t.column :zip_code, :string, :limit => 20
+
+      t.column :effective_date, :timestamp
+      t.column :end_date, :timestamp
+      t.column :latest_version, :boolean
+    end
+
+    create_table(:truncate_test, :force => true) do |t|
+      t.column :x, :string, :limit => 4
+    end
+  end
+
+  def self.down
+    raise ActiveRecord::IrreversibleMigration
+  end
+end

data/db/schema.rb

@@ -0,0 +1,45 @@
+# encoding: UTF-8
+# This file is auto-generated from the current state of the database. Instead
+# of editing this file, please use the migrations feature of Active Record to
+# incrementally modify your database, and then regenerate this schema definition.
+#
+# Note that this schema.rb definition is the authoritative source for your
+# database schema. If you need to create the application database on another
+# system, you should be using db:schema:load, not running all the migrations
+# from scratch. The latter is a flawed and unsustainable approach (the more migrations
+# you'll amass, the slower it'll run and the greater likelihood for issues).
+#
+# It's strongly recommended to check this file into your version control system.
+
+ActiveRecord::Schema.define(:version => 20120229203554) do
+
+  create_table "people", :force => true do |t|
+    t.string "first_name"
+    t.string "last_name"
+    t.string "ssn",        :limit => 64
+  end
+
+  create_table "person_dimension", :force => true do |t|
+    t.string   "first_name",     :limit => 50
+    t.string   "last_name",      :limit => 50
+    t.string   "address",        :limit => 100
+    t.string   "city",           :limit => 50
+    t.string   "state",          :limit => 50
+    t.string   "zip_code",       :limit => 20
+    t.datetime "effective_date"
+    t.datetime "end_date"
+    t.boolean  "latest_version"
+  end
+
+  create_table "places", :force => true do |t|
+    t.text   "address"
+    t.string "city"
+    t.string "state"
+    t.string "country", :limit => 2
+  end
+
+  create_table "truncate_test", :force => true do |t|
+    t.string "x", :limit => 4
+  end
+
+end

data/examples/database.example.yml

@@ -1,16 +1,16 @@
 etl_execution:
-  adapter: mysql
+  adapter: mysql2
   username: root
   host: localhost
   database: etl_execution
   encoding: utf8
 datawarehouse:
-  adapter: mysql
+  adapter: mysql2
   username: root
   host: localhost
   database: datawarehouse_development
 operational:
-  adapter: mysql
+  adapter: mysql2
   username: root
   host: localhost
   database: operational_production

data/lib/etl.rb

@@ -42,6 +42,11 @@ require 'active_support'
 require 'active_record'
 require 'adapter_extensions'
 
+if ActiveSupport::VERSION::STRING >= '3.2'
+  # support for cattr_accessor
+  require 'active_support/core_ext/class/attribute_accessors'
+end
+
 if RUBY_VERSION < '1.9'
   require 'faster_csv'
   CSV = FasterCSV unless defined?(CSV)
@@ -49,6 +54,17 @@ else
   require 'csv'
 end
 
+# patch for https://github.com/activewarehouse/activewarehouse-etl/issues/24
+# allow components to require optional gems
+class Object
+  def optional_require(feature)
+    begin
+      require feature
+    rescue LoadError
+    end
+  end
+end
+
 $:.unshift(File.dirname(__FILE__))
 
 require 'etl/core_ext'