activewarehouse-etl 0.6.1 → 0.7.0

data/CHANGELOG

@@ -67,4 +67,32 @@
 
 0.6.1 - Mar 22, 2007
 * Added support for absolute paths in file sources
-* Added CopyFieldProcessor
+* Added CopyFieldProcessor
+
+0.7 - Apr 8, 2007
+* Job execution is now tracked in a database. This means that ActiveRecord is required regardless
+  of the sources being used in the ETL scripts. An example database configuration for the etl can
+  be found in test/database.example.yml. This file is loaded from either a.) the current working
+  directory or b.) the location specified using the -c command line argument when running the
+  etl command.
+* etl script now supports the following command line arguments:
+** -h or --help: Prints the usage
+** -l or --limit: Specifies a limit for the number of source rows to read, useful for testing
+   your control files before executing a full ETL process
+** -o or --offset: Specified a start offset for reading from the source, useful for testing your
+   control files before executing a full ETL process
+** -c or --config: Specify the database.yml file to configure the ETL execution data store
+** -n or --newlog: Write to the logfile rather than appending to it
+* Database source now supports specifying the select, join and order parts of the query.
+* Database source understands the limit argument specified on the etl command line
+* Added CheckExistProcessor
+* Added CheckUniqueProcessor
+* Added SurrogateKeyProcessor. The SurrogateKey processor should be used in conjunction with the 
+  CheckExistProcessor and CheckUniqueProcessor to provide 
+* Added SequenceProcessor
+* Added OrdinalizeTransform
+* Fixed a bug in the trim transform
+* Sources now provide a trigger file which can be used to indicate that the original source
+  data has been completely extracted to the local file system. This is useful if you need to
+  recover from a failed ETL process.
+* Updated README

data/LICENSE

@@ -0,0 +1,7 @@
+Copyright (c) 2006-2007 Anthony Eden
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README

@@ -1,27 +1,73 @@
-Ruby ETL tool.
+Ruby Extract-Transform-Load (ETL) tool.
 
 == Features
 
 Current supported features:
 
 * ETL Domain Specific Language (DSL) - Control files are specified in a Ruby-based DSL
-* Multiple source types - fixed-width and delimited text files currently supported
+* Multiple source types. Current supported types:
+  * Fixed-width and delimited text files
+  * XML files through SAX
+  * Apache combined log format
 * Multiple destination types - file and database destinations
-* Support for extracting from multiple sources
-* Support for loading to multiple destinations
-* Extensible transformations - comes with built in SHA1 one-way hash example
-* Pre/post processing - export to files and then post process with the bulk import processor for large amounts of data
+* Support for extracting from multiple sources in a single job
+* Support for writing to multiple destinations in a single job
+* A variety of built-in transformations are included:
+  * Date-to-string, string-to-date, string-to-datetime, string-to-timestamp
+  * Type transformation supporting strings, integers, floats and big decimals
+  * Trim
+  * SHA-1
+  * Decode from an external decode file
+  * Default replacement for empty values
+  * Ordinalize
+  * Hierarchy lookup
+  * Foreign key lookup
+  * Ruby blocks
+  * Any custom transformation class
+* A variety of build-in row-level processors
+  * Check exists processor to determine if the record already exists in the destination database
+  * Check unique processor to determine whether a matching record was processed during this job execution
+  * Copy field
+  * Rename field
+  * Hierarchy exploder which takes a tree structure defined through a parent id and explodes it into a hierarchy bridge table
+  * Surrogate key generator including support for looking up the last surrogate key from the target table using a custom query
+  * Sequence generator including support for context-sensitive sequences where the context can be defined as a combination of fields from the source data
+  * New row-level processors can easily be defined and applied
+* Pre-processing
+  * Truncate processor
+* Post-processing
+  * Bulk import using native RDBMS bulk loader tools
 * Virtual fields - Add a field to the destination data which doesn't exist in the source data
+* Built in job and record meta data
+* Support for type 1 and type 2 slowly changing dimensions
+  * Automated effective date and end date time stamping for type 2
+  * CRC checking
 
-
-== Requirements
-
+== Dependencies
+ActiveWarehouse ETL depends on the following gems:
 * ActiveSupport Gem
 * ActiveRecord Gem
 * FasterCSV Gem
+* AdapterExtensions Gem
+
+== Usage
+Once the ActiveWarehouse ETL gem is installed jobs can be invoked using the
+included `etl` script. The etl script includes several command line options
+and can process multiple control files at a time.
+
+Command line options:
+* <tt>--help, -h</tt>: Display the usage message.
+* <tt>--config, -c</tt>: Specify a database.yml configuration file to use.
+* <tt>--limit, -l</tt>: Specify a limit to the number of rows to process. This option is currently only applicable to database sources.
+* <tt>--offset, -o</tt>: Specify the start offset for reading from the source. This option is currently only applicable to database sources.
+* <tt>--newlog, -n</tt>: Instruct the engine to create a new ETL log rather than append to the last ETL log.
+* <tt>--skip-bulk-import, -s</tt>: Skip any bulk imports.
+* <tt>--read-locally</tt>: Read from the local cache (skip source extraction)
 
-== Examples
-Examples can be found in the test directory.
+== Control File Examples
+Control file examples can be found in the test directory.
 
 == Feedback
-This is a work in progress. Comments should be made on the activewarehouse-discuss mailing list at the moment.
+This is a work in progress. Comments should be made on the 
+activewarehouse-discuss mailing list at the moment. Contributions are always
+welcome.

data/Rakefile

@@ -40,6 +40,7 @@ end
 
 PKG_FILES = FileList[
   'CHANGELOG',
+  'LICENSE',
   'README',
   'TODO',
   'Rakefile',
@@ -59,7 +60,7 @@ spec = Gem::Specification.new do |s|
   s.add_dependency('rake',                '>= 0.7.1')
   s.add_dependency('activesupport',       '>= 1.3.1')
   s.add_dependency('activerecord',        '>= 1.14.4')
-  s.add_dependency('fastercsv',           '>= 1.0.0')
+  s.add_dependency('fastercsv',           '>= 1.2.0')
   s.add_dependency('adapter_extensions',  '>= 0.1.0')
 
   s.rdoc_options << '--exclude' << '.'

data/lib/etl.rb

@@ -26,6 +26,8 @@
 #++
 
 require 'logger'
+require 'yaml'
+require 'erb'
 
 require 'rubygems'
 
@@ -59,6 +61,7 @@ require 'etl/http_tools'
 require 'etl/version'
 require 'etl/engine'
 require 'etl/control'
+require 'etl/row'
 require 'etl/parser'
 require 'etl/transform'
 require 'etl/processor'

data/lib/etl/commands/etl.rb

@@ -22,10 +22,43 @@
 #++
 
 require 'benchmark'
+require 'getoptlong'
+
+opts = GetoptLong.new(
+  [ '--help', '-h', GetoptLong::NO_ARGUMENT ],
+  [ '--config', '-c', GetoptLong::REQUIRED_ARGUMENT ],
+  [ '--limit', '-l', GetoptLong::REQUIRED_ARGUMENT ],
+  [ '--offset', '-o', GetoptLong::REQUIRED_ARGUMENT],
+  [ '--newlog', '-n', GetoptLong::NO_ARGUMENT ],
+  [ '--skip-bulk-import', '-s', GetoptLong::NO_ARGUMENT ],
+  [ '--read-locally', GetoptLong::NO_ARGUMENT]
+)
 
 # Print a usage statement
 def usage #:nodoc:
-  puts "Usage: etl ctl_file [ctl_file2 ctl_file3 ...]"
+  puts "Usage: etl ctl_file [ctl_file2 ctl_file3 ...]" # TODO: add the command line options
+end
+
+options = {}
+opts.each do |opt, arg|
+  case opt
+  when '--help'
+    usage
+  when '--config'
+    options[:config] = arg
+  when '--limit'
+    options[:limit] = arg.to_i
+  when '--offset'
+    options[:offset] = arg.to_i
+  when '--newlog'
+    options[:newlog] = true
+  when '--skip-bulk-import'
+    puts "skip bulk import enabled"
+    options[:skip_bulk_import] = true
+  when '--read-locally'
+    puts "read locally enabled"
+    options[:read_locally] = true
+  end
 end
 
 if ARGV.length < 1
@@ -33,6 +66,7 @@ if ARGV.length < 1
 else
   puts "Starting ETL process"
 
+  ETL::Engine.init(options)
   ARGV.each do |f|
     puts "Processing #{f}"
     ETL::Engine.realtime_activity = true

data/lib/etl/control/control.rb

@@ -71,14 +71,13 @@ module ETL #:nodoc:
       
       # Define a transform
       def transform(name, transformer=nil, configuration={}, &block)
-        transforms[name] ||= []
         if transformer
           case transformer
           when String, Symbol
             class_name = "#{transformer.to_s.classify}Transform"
             begin
               transform_class = ETL::Transform.const_get(class_name)
-              transforms[name] << transform_class.new(self, configuration)
+              transforms << transform_class.new(self, name, configuration)
             rescue NameError => e
               raise ControlError, "Unable to find transformer #{class_name}: #{e}"
             end
@@ -86,13 +85,15 @@ module ETL #:nodoc:
             #transformer.class.inspect
             if transformer.is_a?(ETL::Transform::Transform)
               Engine.logger.debug "Adding transformer #{transformer.inspect} for field #{name}"
-              transforms[name]  << transformer
+              t = transformer.dup
+              t.name = name
+              transforms << t 
             else
               raise ControlError, "Transformer must be a String, Symbol or Transform instance"
             end
           end
         elsif block_given?
-          transforms[name] << block
+          transforms << ETL::Transform::BlockTransform.new(self, name, :block => block)
         else
           raise ControlError, "Either a transformer or a block must be specified"
         end
@@ -103,6 +104,16 @@ module ETL #:nodoc:
         control.transforms
       end
       
+      # Rename the source field to the destination field
+      def rename(source, destination)
+        after_read :rename, :source => source, :dest => destination
+      end
+      
+      # Copy the source field to the destination field
+      def copy(source, destination)
+        after_read :copy_field, :source => source, :dest => destination
+      end
+      
       protected
       # This method is used to define a processor and insert into the specified processor
       # collection.
@@ -244,9 +255,9 @@ module ETL #:nodoc:
       end
       
       # Get the transforms with the specified name
-      def transform(name)
-        transforms[name] ||= []
-      end
+      # def transform(name)
+#         transforms[name] ||= []
+#       end
       
       def after_read_processors
         @after_read_processors ||= []
@@ -267,9 +278,9 @@ module ETL #:nodoc:
         @post_processors ||= []
       end
       
-      # Get a map of all transforms for this control
+      # Get an Array of all transforms for this control
       def transforms
-        @transforms ||= {}
+        @transforms ||= []
       end
       
       # Get the error threshold. Defaults to 100.

data/lib/etl/control/destination.rb

@@ -26,8 +26,8 @@ module ETL #:nodoc:
       class << self
         # Get the destination class for the specified name.
         # 
-        # For example if name is :database or 'database' then the DatabaseDestination class 
-        # is returned
+        # For example if name is :database or 'database' then the 
+        # DatabaseDestination class is returned
         def class_for_name(name)
           ETL::Control.const_get("#{name.to_s.classify}Destination")
         end
@@ -42,13 +42,14 @@ module ETL #:nodoc:
       #
       # Options:
       # * <tt>:buffer_size</tt>: The output buffer size (default 1000 records)
-      # * <tt>:condition</tt>: A conditional proc that must return true for the row to be written
+      # * <tt>:condition</tt>: A conditional proc that must return true for the 
+      #   row to be written
       # * <tt>:append_rows</tt>: An array of rows to append
       def initialize(control, configuration, mapping)
         @control = control
         @configuration = configuration
         @mapping = mapping
-        @buffer_size = configuration[:buffer_size] ||= 1000
+        @buffer_size = configuration[:buffer_size] ||= 100
         @condition = configuration[:condition]
         @append_rows = configuration[:append_rows]
       end
@@ -61,7 +62,7 @@ module ETL #:nodoc:
       # Write the given row
       def write(row)
         if @condition.nil? || @condition.call(row)
-          buffer << row
+          process_change(row)
         end
         flush if buffer.length >= buffer_size
       end
@@ -105,8 +106,9 @@ module ETL #:nodoc:
         order
       end
       
-      # Return true if the row is allowed. The row will not be allowed if the :unique option is specified
-      # in the configuration and the compound key already exists
+      # Return true if the row is allowed. The row will not be allowed if the
+      # :unique option is specified in the configuration and the compound key 
+      # already exists
       def row_allowed?(row)
         if unique
           key = (unique.collect { |k| row[k] }).join('|')
@@ -116,15 +118,160 @@ module ETL #:nodoc:
         return true
       end
       
-      # Get a hash of compound key contraints. This is used to determine if a row can be written when the
-      # unique option is specified
+      # Get a hash of compound key contraints. This is used to determine if a
+      # row can be written when the unique option is specified
       def compound_key_constraints
         @compound_key_constraints ||= {}
       end
       
-      # Add any virtual fields to the row. Virtual rows will get their value from one of the following:
-      # * If the mapping is a Class, then an object which implements the next method
-      # * If the mapping is a Symbol, then the XGenerator where X is the classified symbol
+      # Return fields which are Slowly Changing Dimension fields. Return nil 
+      # by default.
+      def scd_fields
+        @scd_fields ||= configuration[:scd_fields]
+      end
+      
+      def scd?
+        !configuration[:scd].nil?
+      end
+      
+      def scd_type
+        scd? ? configuration[:scd][:type] : nil
+      end
+      
+      # Get the Slowly Changing Dimension effective date field. Defaults to
+      # 'effective_date'.
+      def scd_effective_date_field
+        configuration[:scd][:effective_date_field] || :effective_date if scd?
+      end
+      
+      # Get the Slowly Changing Dimension end date field. Defaults to 
+      # 'end_date'.
+      def scd_end_date_field
+        configuration[:scd][:end_date_field] || :end_date if scd?
+      end
+      
+      # Return the natural key field name, defaults to :id
+      def natural_key
+        @natural_key ||= determine_natural_key
+      end
+      
+      # Get the dimension table if specified
+      def dimension_table
+        configuration[:scd][:dimension_table] if scd?
+      end
+      
+      # Process a row to determine the change type
+      def process_change(row)
+        return unless row
+        
+        # Change processing can only occur if the natural key exists in the row 
+        supports_change = true
+        natural_key.each do |key| 
+          unless row.has_key?(key)
+            buffer << row
+            return
+          end
+        end
+        
+        ETL::Engine.logger.debug "checking scd fields"
+        s = String.new
+        if scd_fields
+          scd_fields.each { |f| s << row[f].to_s }
+        else
+          row.each { |key,value| s << value.to_s }
+        end
+      
+        # apply the CRC to 's' and see if it matches the last 
+        # ETL::Execution::Record with the samenatural key. If they match then 
+        # throw away this row (no need to process). If they do not match then 
+        # the record is an 'update'. If the record doesn't exist then it is an
+        # 'insert'
+        nk = natural_key.collect{|k|row[k]}.join('|')
+        require 'zlib'
+        crc = Zlib.crc32(s)
+        record = ETL::Execution::Record.find_by_control_file_and_natural_key(control.file, nk)
+        
+        timestamp = Time.now
+        
+        ETL::Engine.logger.debug "checking record change type"
+        if record
+          if record.crc != crc.to_s
+            # SCD Type 1: only the new row should be added
+            # SCD Type 2: both an old and new row should be added
+            # SCD Type 3: not supported
+            ETL::Engine.logger.debug "CRC does not match"
+            
+            if scd_type == 2
+              ETL::Engine.logger.debug "type 2 SCD"
+              q = "SELECT * FROM #{dimension_table} WHERE "
+              q << natural_key.collect { |nk| "#{nk} = '#{row[nk]}'" }.join(" AND ")
+              #puts "looking for original record"
+              result = ActiveRecord::Base.connection.select_one(q)
+              if result
+                #puts "Result: #{result.inspect}"
+                original_record = ETL::Row[result.symbolize_keys!]
+                original_record[scd_end_date_field] = timestamp
+                ETL::Engine.logger.debug "writing original record"
+                buffer << original_record
+              end
+              
+              row[scd_effective_date_field] = timestamp
+              row[scd_end_date_field] = '9999-12-31 00:00:00'
+            elsif scd_type == 1
+              ETL::Engine.logger.debug "type 1 SCD"
+            else
+              ETL::Engine.logger.debug "SCD not specified"
+            end
+            
+            ETL::Engine.logger.debug "writing new record"
+            buffer << row
+          else
+            ETL::Engine.logger.debug "CRC matches, skipping"
+            
+            q = "SELECT * FROM #{dimension_table} WHERE "
+            q << natural_key.collect { |nk| "#{nk} = '#{row[nk]}'" }.join(" AND ")
+            result = ActiveRecord::Base.connection.select_one(q)
+            if result
+              buffer << ETL::Row[result.symbolize_keys!]
+            else
+              # The record never made it into the database, so add the effective and end date
+              # and add it into the bulk load file
+              row[scd_effective_date_field] = timestamp
+              row[scd_end_date_field] = '9999-12-31 00:00:00'
+              buffer << row
+            end
+          end
+        else
+          ETL::Engine.logger.debug "record never loaded"
+          # Set the effective and end date fields
+          if scd_type == 2
+            row[scd_effective_date_field] = timestamp
+            row[scd_end_date_field] = '9999-12-31 00:00:00'
+          end
+          
+          # Write the row
+          buffer << row
+          
+          # Record the record
+          if ETL::Engine.job # only record the execution if there is a job
+            ETL::Execution::Record.create!(
+              :control_file => control.file,
+              :natural_key => nk,
+              :crc => crc,
+              :job_id => ETL::Engine.job.id
+            )
+          end
+        end
+      rescue => e
+        puts e
+      end
+      
+      # Add any virtual fields to the row. Virtual rows will get their value 
+      # from one of the following:
+      # * If the mapping is a Class, then an object which implements the next 
+      #   method
+      # * If the mapping is a Symbol, then the XGenerator where X is the 
+      #   classified symbol
       # * If the mapping is a Proc, then it will be called with the row
       # * Otherwise the value itself will be assigned to the field
       def add_virtuals!(row)
@@ -146,6 +293,20 @@ module ETL #:nodoc:
           end
         end
       end
+      
+      private
+      # Determine the natural key. This method will always return an array
+      # of symbols. The default value is [:id].
+      def determine_natural_key
+        case configuration[:natural_key]
+        when Array
+          configuration[:natural_key].collect(&:to_sym)
+        when String, Symbol
+          [configuration[:natural_key].to_sym]
+        else
+          [:id]
+        end
+      end
     end
   end
 end