arc-furnace 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 38e438f57e6350ddce2937cf2ef03d6d93ea20a8
+  data.tar.gz: 17ee74aba7edef53647c2a2e4efe6b4d3bda477f
+SHA512:
+  metadata.gz: ee8d88b55edd486dbd496d2831096ed53eaad124e6043629747b9503bd6473267c1f18086edae0425152b08638fc7ed4961cfecc79b72a92d709768c49316b25
+  data.tar.gz: 39114078eab439a660c83479b9251b0e9f1690aa8e0110b51a4dc061aee5f7d64421189c69f0705349d95339b14e7e5b118124b1822a7c3e747446313c585ec7

data/.gitignore

@@ -0,0 +1,9 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/

data/.rspec

@@ -0,0 +1,2 @@
+--format documentation
+--color

data/.travis.yml

@@ -0,0 +1,6 @@
+language: ruby
+sudo: false
+rvm:
+  - 2.1.6
+  - 2.2.3
+  - jruby-9000

data/Gemfile

@@ -0,0 +1,3 @@
+source 'https://rubygems.org'
+
+gemspec

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2015 Daniel Spangenberger
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,127 @@
+# ArcFurnace
+[![Build Status](https://travis-ci.org/salsify/arc-furnace.svg?branch=master)][travis]
+
+[travis]: http://travis-ci.org/salsify/arc-furnace
+
+ArcFurnace melts, melds, and transforms your scrap data into perfectly crafted data for ingest into applications,
+analysis, or whatnot. ArcFurnace simplifies simple ETL (Extract, Transform, Load) tasks for small to medium sets of data
+using a programmatic DSL interface. Here's an example:
+
+```ruby
+class Transform < ArcFurnace::Pipeline
+
+    source :marketing_info_csv, type: ArcFurnace::CSVSource, params: { filename: :marketing_filename }
+
+    transform :marketing_info_source, params: { source: :marketing_info_csv } do |row|
+      row.delete('Name')
+      row
+    end
+
+    source :product_attributes,
+           type: ArcFurnace::MultiCSVSource,
+           params: { filenames: :product_attribute_filenames }
+
+    hash_node :marketing_info,
+              params: {
+                  key_column: :primary_key,
+                  source: :marketing_info_source
+              }
+
+    outer_join :join_results,
+               params: {
+                   source: :product_attributes,
+                   hash: :marketing_info
+               }
+
+    sink type: ArcFurnace::AllFieldsCSVSink,
+         source: :join_results,
+         params: { filename: :destination_name }
+
+end
+```
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'arc-furnace', github: 'salsify/arc-furnace'
+```
+
+And then execute:
+
+    $ bundle
+
+## Usage
+
+ArcFurnace provides a few concepts useful to extracting and transforming data.
+
+### Node Types Available
+
+#### Pipelines
+
+Pipelines define a a complete transformation and define a directed, acyclic graph of
+operations that define how data is transformed. Each type of node in a `Pipeline` is defined below, but
+a Pipelines defines the network of nodes that transform data.
+
+#### Sources
+
+A `Source` provides values to a `Pipeline`. A `Pipeline` may have many sources. Essentially, any nodes that
+require a stream of data (`Hash`, `Transform`, `Join`, `Sink`) will have one.
+
+#### Hashes
+
+A `Hash` provides indexed access to a `Source` but pre-computing the index based on a key. The processing happens during the 
+prepare stage of pipeline processing. Hashes have a simple interface, `#get(primary_key)`, to requesting data. Hashes
+are almost exclusively used as inputs to one side of joins.
+
+#### Joins
+
+An `InnerJoin` or an `OuterJoin` join two sources of data (one must be a `Hash`) based upon a key. By default the join
+key is the key that the hash was rolled-up on, however, the `key_column` option on both `InnerJoin` and `OuterJoin`
+may override this. Note the default join is an inner join, which will drop source rows if the hash does not contain
+a matching row.
+
+#### Transforms
+
+A `Transform` acts as a source, however, takes a source as an input and transforms each input. The `BlockTransform` and
+associated sugar in the `transform` method of `Pipeline` make this very easy (see the example above).
+
+#### Unfolds
+
+An `Unfold` acts as a source, however, takes a source as an input and produces multiple rows for that source as an output.
+A common case for this is splitting rows into multiple rows depending upon their keys. The `BlockTransform` and associated
+sugar in the `unfold` method of `Pipeline` make this fiarly easy (see `pipeline_spec.rb`).
+
+#### Sinks
+
+Each `Pipeline` has a single sink. Pipelines must produce data somewhere, and that data goes to a sink. Sinks
+subscribe to the `#row(hash)` interace--each output row is passed to this method for handling.
+
+### General pipeline development process
+
+1. Define a source. Choose an existing `Source` implementation in this library (`CSVSource` or `ExcelSource`), 
+   extend the `EnumeratorSource`, or implement the `row()` method for a new source.
+2. Define any transformations, or joins. This may cause you to revisit #1.
+3. Define the sink. This is generally custom, or, may be one of the provided `CSVSink` types.
+4. Roll it together in a `Pipeline`.
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release` to create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## TODOs
+
+1. Add a `filter` node and implementation to `Pipeline`
+2. Add examples for `ErrorHandler` interface.
+3. Add sugar to define a `BlockTransform` on a `Source` definition in a `Pipeline`.
+
+## Contributing
+
+1. Fork it ( https://github.com/[my-github-username]/arc-furnace/fork )
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create a new Pull Request

data/Rakefile

@@ -0,0 +1,9 @@
+require 'bundler/setup'
+Bundler::GemHelper.install_tasks
+
+require 'rspec/core/rake_task'
+RSpec::Core::RakeTask.new(:spec) do |task|
+  task.verbose = false
+end
+
+task default: :spec

data/arc-furnace.gemspec

@@ -0,0 +1,30 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'arc-furnace/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "arc-furnace"
+  spec.version       = ArcFurnace::VERSION
+  spec.authors       = ["Daniel Spangenberger", "Brian Tenggren"]
+  spec.email         = ["dan@salsify.com"]
+
+  spec.summary       = %q{Melds and transforms data from multiple sources into a single stream}
+  spec.description   = %q{An ETL library for Ruby that performs the basic actions of ETL: extract, transform, and load. Easily extensible.}
+  spec.homepage      = "http://github.com/salsify/arc-furnace"
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+  spec.bindir        = "exe"
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency 'msgpack', '~> 0.6'
+  spec.add_dependency 'activesupport', '>= 3.2'
+  spec.add_dependency 'eigenclass', '~> 2'
+  spec.add_dependency 'roo', '>= 2.1'
+
+  spec.add_development_dependency 'rake', '~> 10.0'
+  spec.add_development_dependency 'rspec', '~> 3'
+  spec.add_development_dependency 'ice_nine', '>= 0.11'
+end

data/lib/arc-furnace.rb

@@ -0,0 +1,12 @@
+require 'arc-furnace/version'
+
+require 'active_support'
+require 'active_support/core_ext'
+
+require 'arc-furnace/csv_to_hash_with_duplicate_headers'
+require 'arc-furnace/private_attr'
+
+require 'arc-furnace/dsl'
+require 'arc-furnace/error_handler'
+require 'arc-furnace/nodes'
+require 'arc-furnace/pipeline'

data/lib/arc-furnace/abstract_join.rb

@@ -0,0 +1,53 @@
+require 'arc-furnace/source'
+
+module ArcFurnace
+  class AbstractJoin < Source
+    private_attr_reader :hash, :source, :key_column
+    attr_reader :value
+
+    # The source is a source, the hash is a hash, and one can optionally
+    # pass the key column to get the primary key for each source entity, the
+    # default is equijoin semantics--the key of the hash is used.
+    def initialize(source: , hash:, key_column: nil)
+      if source.is_a?(::ArcFurnace::Source) && hash.is_a?(::ArcFurnace::Hash)
+        @hash = hash
+        @source = source
+        @key_column = key_column || hash.key_column
+      else
+        raise 'Must be passed one Hash and one Source!'
+      end
+    end
+
+    def prepare
+      advance
+    end
+
+    def advance
+      raise "Unimplemented!"
+    end
+
+    delegate empty?: :source
+
+    protected
+
+    def merge_source_row(source_row)
+      key = source_row[key_column]
+      if key
+        if hash_value = hash.get(key)
+          hash_value = hash_value.deep_dup
+          source_row.each do |key, value|
+            hash_value[key] = value
+          end
+          @value = hash_value
+          true
+        else
+          error_handler.missing_hash_key(source_row: source_row, key: key, node_id: node_id)
+          false
+        end
+      else
+        error_handler.missing_join_key(source_row: source_row, node_id: node_id)
+      end
+    end
+
+  end
+end

data/lib/arc-furnace/all_fields_csv_sink.rb

@@ -0,0 +1,68 @@
+require 'arc-furnace/sink'
+require 'msgpack'
+
+module ArcFurnace
+  class AllFieldsCSVSink < Sink
+    private_attr_reader :csv, :fields, :tmp_file, :packer, :fields, :field_mappings
+
+    def initialize(filename: , encoding: 'UTF-8')
+      @tmp_file = Tempfile.new('intermediate_results', encoding: 'binary')
+      @packer = MessagePack::Packer.new(tmp_file)
+      @csv = CSV.open(filename, 'wb', encoding: encoding, headers: true)
+      @fields = {}
+    end
+
+    def prepare(fields = nil)
+
+    end
+
+    def finalize
+      packer.flush
+      tmp_file.rewind
+
+      write_header_row!
+
+      unpacker = MessagePack::Unpacker.new(tmp_file)
+      unpacker.each do |hash|
+        write_row(hash)
+      end
+
+      csv.close
+    end
+
+    def row(hash)
+      update_field_counts(hash)
+      packer.write(hash)
+    end
+
+    private
+
+    def write_header_row!
+      header_row = []
+      fields.each do |key, count|
+        count.times { header_row << key }
+      end
+      csv << header_row
+    end
+
+    def write_row(hash)
+      row = []
+      fields.each do |key, count|
+        values = Array.wrap(hash[key])
+        (values.slice(0, count) || []).each do |value|
+          row << value
+        end
+        (count - values.length).times { row << nil }
+      end
+      csv << row
+    end
+
+    def update_field_counts(hash)
+      hash.each do |key, values|
+        value_count = Array.wrap(values).size
+        existing_value_count = fields[key] || 0
+        fields[key] = value_count if value_count > existing_value_count
+      end
+    end
+  end
+end

data/lib/arc-furnace/binary_key_merging_hash.rb

@@ -0,0 +1,38 @@
+require 'arc-furnace/hash'
+
+module ArcFurnace
+  # This allows one to merge multiple rows into one such as:
+  #   key, attribute, value
+  #     1, value1, foo
+  #     1, value1, bar
+  #     1, value2, baz
+  # Results in:
+  #   1 => {  value1 => [foo, bar], value2 => baz }
+  class BinaryKeyMergingHash < ::ArcFurnace::Hash
+    private_attr_reader :source, :hash, :secondary_key, :value_key
+
+    def initialize(source: , primary_key:, secondary_key:, value_key:)
+      super(source: source, key_column: primary_key)
+      @secondary_key = secondary_key
+      @value_key = value_key
+    end
+
+    def prepare
+      loop do
+        break if source.empty?
+        row = source.row
+        row_key = row[key_column]
+        second_key = row[secondary_key]
+        value = row[value_key]
+        if row_key && second_key && value
+          row_entry = hash[row_key] ||= {}
+          value_arr = row_entry[second_key] ||= []
+          value_arr.concat(Array.wrap(value))
+        else
+          error_handler.missing_primary_key(source_row: row, node_id: node_id)
+        end
+      end
+    end
+
+  end
+end

data/lib/arc-furnace/block_transform.rb

@@ -0,0 +1,18 @@
+require 'arc-furnace/transform'
+
+module ArcFurnace
+  class BlockTransform < Transform
+    private_attr_reader :block
+
+    def initialize(source:, block:)
+      raise 'Must specify a block' if block.nil?
+      @block = block
+      super(source: source)
+    end
+
+    def transform(row)
+      block.call(row)
+    end
+
+  end
+end

data/lib/arc-furnace/block_unfold.rb

@@ -0,0 +1,18 @@
+require 'arc-furnace/unfold'
+
+module ArcFurnace
+  class BlockUnfold < Unfold
+    private_attr_reader :block
+
+    def initialize(source:, block:)
+      raise 'Must specify a block' if block.nil?
+      @block = block
+      super(source: source)
+    end
+
+    def unfold(row)
+      block.call(row)
+    end
+
+  end
+end