activewarehouse-etl 0.9.5.rc1 → 1.0.0.rc1

data/README

@@ -1,101 +0,0 @@
-Ruby Extract-Transform-Load (ETL) tool.
-
-== Requirements
-
-* Ruby 1.8.5 or higher
-* Rubygems
-
-== Online Documentation
-
-Available at https://github.com/activewarehouse/activewarehouse-etl/wiki
-
-== Features
-
-Current supported features:
-
-* ETL Domain Specific Language (DSL) - Control files are specified in a Ruby-based DSL
-* Multiple source types. Current supported types:
-  * Fixed-width and delimited text files
-  * XML files through SAX
-  * Apache combined log format
-* Multiple destination types - file and database destinations
-* Support for extracting from multiple sources in a single job
-* Support for writing to multiple destinations in a single job
-* A variety of built-in transformations are included:
-  * Date-to-string, string-to-date, string-to-datetime, string-to-timestamp
-  * Type transformation supporting strings, integers, floats and big decimals
-  * Trim
-  * SHA-1
-  * Decode from an external decode file
-  * Default replacement for empty values
-  * Ordinalize
-  * Hierarchy lookup
-  * Foreign key lookup
-  * Ruby blocks
-  * Any custom transformation class
-* A variety of build-in row-level processors
-  * Check exists processor to determine if the record already exists in the destination database
-  * Check unique processor to determine whether a matching record was processed during this job execution
-  * Copy field
-  * Rename field
-  * Hierarchy exploder which takes a tree structure defined through a parent id and explodes it into a hierarchy bridge table
-  * Surrogate key generator including support for looking up the last surrogate key from the target table using a custom query
-  * Sequence generator including support for context-sensitive sequences where the context can be defined as a combination of fields from the source data
-  * New row-level processors can easily be defined and applied
-* Pre-processing
-  * Truncate processor
-* Post-processing
-  * Bulk import using native RDBMS bulk loader tools
-* Virtual fields - Add a field to the destination data which doesn't exist in the source data
-* Built in job and record meta data
-* Support for type 1 and type 2 slowly changing dimensions
-  * Automated effective date and end date time stamping for type 2
-  * CRC checking
-
-== Dependencies
-ActiveWarehouse ETL depends on the following gems:
-* ActiveSupport Gem
-* ActiveRecord Gem
-* FasterCSV Gem
-* AdapterExtensions Gem
-
-== Usage
-Once the ActiveWarehouse ETL gem is installed jobs can be invoked using the
-included `etl` script. The etl script includes several command line options
-and can process multiple control files at a time.
-
-Command line options:
-* <tt>--help, -h</tt>: Display the usage message.
-* <tt>--config, -c</tt>: Specify a database.yml configuration file to use.
-* <tt>--limit, -l</tt>: Specify a limit to the number of rows to process. This option is currently only applicable to database sources.
-* <tt>--offset, -o</tt>: Specify the start offset for reading from the source. This option is currently only applicable to database sources.
-* <tt>--newlog, -n</tt>: Instruct the engine to create a new ETL log rather than append to the last ETL log.
-* <tt>--skip-bulk-import, -s</tt>: Skip any bulk imports.
-* <tt>--read-locally</tt>: Read from the local cache (skip source extraction)
-
-== Control File Examples
-Control file examples can be found in the examples directory.
-
-== Running Tests
-
-Current state:
-- 11 failures on MySQL
-- 1 failure on Postgres
-
-The tests require:
-- gem install shoulda
-- gem install flexmock
-- gem install pg (if you want to run the tests on pg)
-- gem install spreadsheet
-- gem install tmail
-
-The tests subfolder contains examples database.yml for mysql and postgres.
-
-To run the tests:
-- rake test DB=postgresql (for postgres)
-- otherwise just rake test
-
-== Feedback
-This is a work in progress. Comments should be made on the 
-activewarehouse-discuss mailing list at the moment. Contributions are always
-welcome.

data/active_support_logger.patch

@@ -1,78 +0,0 @@
-Index: lib/active_support/clean_logger.rb
-===================================================================
---- lib/active_support/clean_logger.rb	(revision 5963)
-+++ lib/active_support/clean_logger.rb	(working copy)
-@@ -1,10 +1,21 @@
- require 'logger'
- require File.dirname(__FILE__) + '/core_ext/class/attribute_accessors'
- 
--class Logger #:nodoc:
-+# Extensions to the built in Ruby logger.
-+#
-+# If you want to use the default log formatter as defined in the Ruby core, then you 
-+# will need to set the formatter for the logger as in:
-+#
-+#   logger.formatter = Formatter.new
-+#
-+# You can then specify the datetime format, for example:
-+#
-+#   logger.datetime_format = "%Y-%m-%d"
-+class Logger
-+  # Set to false to disable the silencer
-   cattr_accessor :silencer
-   self.silencer = true
--
-+  
-   # Silences the logger for the duration of the block.
-   def silence(temporary_level = Logger::ERROR)
-     if silencer
-@@ -18,6 +29,35 @@
-       yield self
-     end
-   end
-+  
-+  alias :old_datetime_format= :datetime_format=
-+  # Logging date-time format (string passed to +strftime+). Ignored if the formatter
-+  # does not respond to datetime_format=.
-+  def datetime_format=(datetime_format)
-+    formatter.datetime_format = datetime_format if formatter.respond_to?(:datetime_format=)
-+  end
-+  
-+  alias :old_datetime_format :datetime_format
-+  # Get the logging datetime format. Returns nil if the formatter does not support
-+  # datetime formatting.
-+  def datetime_format
-+    formatter.datetime_format if formatter.respond_to?(:datetime_format)
-+  end
-+  
-+  alias :old_formatter :formatter
-+  # Get the current formatter. The default formatter is a SimpleFormatter which only
-+  # displays the log message
-+  def formatter
-+    @formatter ||= SimpleFormatter.new
-+  end
-+  
-+  # Simple formatter which only displays the message.
-+  class SimpleFormatter < Logger::Formatter
-+    # This method is invoked when a log event occurs
-+    def call(severity, timestamp, progname, msg)
-+      "#{msg}\n"
-+    end
-+  end
- 
-   private
-     alias old_format_message format_message
-@@ -28,11 +68,11 @@
-     # with Logger from 1.8.3 and vice versa.
-     if method_defined?(:formatter=)
-       def format_message(severity, timestamp, progname, msg)
--        "#{msg}\n"
-+        formatter.call(severity, timestamp, progname, msg)
-       end
-     else
-       def format_message(severity, timestamp, msg, progname)
--        "#{msg}\n"
-+        formatter.call(severity, timestamp, progname, msg)
-       end
-     end
- end

data/test-matrix.yml

@@ -1,10 +0,0 @@
-rvm:
-  - 1.8.7
-  - 1.9.2
-#  - jruby-1.6.2
-rails:
-  - 2.3.x
-  - 3.0.x
-database:
-  - mysql
-  - postgresql

data/test/config/Gemfile.rails-2.3.x.lock

@@ -1,38 +0,0 @@
-GEM
-  remote: http://rubygems.org/
-  specs:
-    activerecord (2.3.11)
-      activesupport (= 2.3.11)
-    activesupport (2.3.11)
-    fastercsv (1.5.4)
-    flexmock (0.9.0)
-    mysql (2.8.1)
-    net-sftp (2.0.5)
-      net-ssh (>= 2.0.9)
-    net-ssh (2.1.4)
-    pg (0.11.0)
-    rdoc (3.6.1)
-    ruby-ole (1.2.11.1)
-    shoulda (2.11.3)
-    spreadsheet (0.6.5.4)
-      ruby-ole (>= 1.0)
-    tmail (1.2.7.1)
-    zip (2.0.2)
-
-PLATFORMS
-  java
-  ruby
-
-DEPENDENCIES
-  activerecord (= 2.3.11)
-  activesupport (= 2.3.11)
-  fastercsv (= 1.5.4)
-  flexmock (= 0.9.0)
-  mysql (= 2.8.1)
-  net-sftp (= 2.0.5)
-  pg (= 0.11.0)
-  rdoc
-  shoulda (= 2.11.3)
-  spreadsheet (= 0.6.5.4)
-  tmail (= 1.2.7.1)
-  zip (= 2.0.2)

data/test/config/Gemfile.rails-3.0.x.lock

@@ -1,49 +0,0 @@
-GEM
-  remote: http://rubygems.org/
-  specs:
-    activemodel (3.0.7)
-      activesupport (= 3.0.7)
-      builder (~> 2.1.2)
-      i18n (~> 0.5.0)
-    activerecord (3.0.7)
-      activemodel (= 3.0.7)
-      activesupport (= 3.0.7)
-      arel (~> 2.0.2)
-      tzinfo (~> 0.3.23)
-    activesupport (3.0.7)
-    arel (2.0.10)
-    builder (2.1.2)
-    fastercsv (1.5.4)
-    flexmock (0.9.0)
-    i18n (0.5.0)
-    mysql (2.8.1)
-    net-sftp (2.0.5)
-      net-ssh (>= 2.0.9)
-    net-ssh (2.1.4)
-    pg (0.11.0)
-    rdoc (3.6.1)
-    ruby-ole (1.2.11.1)
-    shoulda (2.11.3)
-    spreadsheet (0.6.5.4)
-      ruby-ole (>= 1.0)
-    tmail (1.2.7.1)
-    tzinfo (0.3.27)
-    zip (2.0.2)
-
-PLATFORMS
-  java
-  ruby
-
-DEPENDENCIES
-  activerecord (= 3.0.7)
-  activesupport (= 3.0.7)
-  fastercsv (= 1.5.4)
-  flexmock (= 0.9.0)
-  mysql (= 2.8.1)
-  net-sftp (= 2.0.5)
-  pg (= 0.11.0)
-  rdoc
-  shoulda (= 2.11.3)
-  spreadsheet (= 0.6.5.4)
-  tmail (= 1.2.7.1)
-  zip (= 2.0.2)

data/test/config/common.rb

@@ -1,21 +0,0 @@
-def common_gemfile(rails_version)
-  source :rubygems
-
-  # using explicit versions for the gems to avoid any weirdness later on
-  gem "activesupport", rails_version
-  gem "activerecord", rails_version
-
-  gem "fastercsv", "1.5.4"
-  gem "spreadsheet", "0.6.5.4"
-  gem "tmail", "1.2.7.1"
-  gem "net-sftp", "2.0.5"
-  gem "zip", "2.0.2"
-
-  gem "shoulda", "2.11.3"
-  gem "flexmock", "0.9.0"
-
-  gem "mysql", "2.8.1"
-  gem "pg", "0.11.0"
-
-  gem "rdoc"
-end

data/test/connection/mysql/connection.rb

@@ -1,9 +0,0 @@
-print "Using native MySQL\n"
-
-puts "Resetting database"
-conn = ETL::Engine.connection(:data_warehouse)
-conn.recreate_database(conn.current_database)
-conn.reconnect!
-lines = open(File.join(File.dirname(__FILE__), 'schema.sql')).readlines
-lines.join.split(';').each { |line| conn.execute(line) }
-conn.disconnect!

data/test/connection/mysql/schema.sql

@@ -1,36 +0,0 @@
-drop table if exists people;
-create table people (
-  id int not null primary key,
-  first_name char(255) not null,
-  last_name char(255) not null,
-  ssn char(64) not null
-);
-drop table if exists places;
-create table places (
-	address text,
-	city char(255),
-	state char(255),
-	country char(2)
-);
-
-drop table if exists person_dimension;
-create table person_dimension (
-  id int not null primary key,
-  first_name char(50),
-  last_name char(50),
-  address char(100),
-  city char(50),
-  state char(50),
-  zip_code char(20),
-  effective_date datetime,
-  end_date datetime,
-  latest_version boolean not null
-);
-
-drop table if exists truncate_test;
-create table truncate_test (
-	x char(4)
-);
-insert into truncate_test (x) values ('a');
-insert into truncate_test (x) values ('b');
-insert into truncate_test (x) values ('c');

data/test/connection/postgresql/connection.rb

@@ -1,13 +0,0 @@
-print "Using PostgreSQL\n"
-
-puts "Resetting database"
-conn = ETL::Engine.connection(:data_warehouse)
-
-lines = open(File.join(File.dirname(__FILE__), 'schema.sql')).readlines
-lines.join.split(';').each_with_index do |line, index|
-  begin
-    conn.execute(line)
-  rescue => e
-    puts "failed to load line #{index}: #{e}"
-  end
-end

data/test/connection/postgresql/schema.sql

@@ -1,39 +0,0 @@
-drop table people;
-create table people (
-  id SERIAL PRIMARY KEY,
-  first_name character varying(255) not null,
-  /* null below allowed for bulk_import_with_empties.txt test */
-  last_name character varying(255) null,
-  ssn character varying(64) not null
-);
-
-drop table places;
-create table places (
-  id SERIAL PRIMARY KEY,
-	address text,
-	city character varying(255),
-	state character varying(255),
-	country character varying(2)
-);
-
-drop table person_dimension;
-create table person_dimension (
-  id SERIAL PRIMARY KEY,
-  first_name character varying(50),
-  last_name character varying(50),
-  address character varying(100),
-  city character varying(50),
-  state character varying(50),
-  zip_code character varying(20),
-  effective_date timestamp without time zone,
-  end_date timestamp without time zone,
-  latest_version boolean not null
-);
-
-drop table truncate_test;
-create table truncate_test (
-	x character varying(4)
-);
-insert into truncate_test (x) values ('a');
-insert into truncate_test (x) values ('b');
-insert into truncate_test (x) values ('c');

data/test/vendor/adapter_extensions-0.5.0/CHANGELOG

@@ -1,26 +0,0 @@
-0.1.0 - March 5, 2007
-* Initial release
-
-0.1.1 - March 5, 2007
-* Bug fixes
-
-0.1.2 - March 5, 2007
-* Bug fixes
-
-0.2.0 - March 6, 2007
-* SQL Server adapter included (Seth Ladd)
-
-0.3.0 - March 8, 2007
-* PostgreSQL adapter included
-* Added tests for bulk loading
-* bulk_load method now handles table missing and file missing as error cases
-
-0.3.1 - May 4, 2007
-* Added support for modifying SELECT statements to add an INSERT INTO.
-
-0.4 - September 17, 2007
-* Added copy_table method that can copy the structure and data from one table to another. Currently implemented in MySQL (tested), PostgreSQL (tested) and SQL Server adapters (untested).
-* Added support for SELECT..INTO for PostgreSQL.
-
-0.5 - 
-* Updated dependencies for gem to current versions of ActiveRecord, ActiveSupport and Rake. May not be compatible with Rails versions less than 2.x.

data/test/vendor/adapter_extensions-0.5.0/LICENSE

@@ -1,16 +0,0 @@
-Copyright (c) 2007 Anthony Eden
-
-Permission is hereby granted, free of charge, to any person obtaining a copy of this software and 
-associated documentation files (the "Software"), to deal in the Software without restriction, including 
-without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 
-copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the 
-following conditions:
-
-The above copyright notice and this permission notice shall be included in all copies or substantial 
-portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT 
-LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN 
-NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, 
-WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 
-SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.