retl 0.0.3

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 65892cccb09b0ee2bde26342b072f3c405b4b735
+  data.tar.gz: b56d3d24285ce3c40aa5e8e98924fe2cf0a09341
+SHA512:
+  metadata.gz: 50953eae75e94eb9383320c56268c0a18e3d1289d234509e13a90eb4c8d43ae2686c6adf6f45f45aed880680b0dd389f2a532a22555ed3bb140aa79a9f8f27cc
+  data.tar.gz: 7cd0b87b33d5810820e978722f3c2e3b63f160a01b65ba4105e4f800135b43a2ec68d65eacf034d78542a8da9de6d3aca263a19df9dced3d600ef304952f9940

data/.gitignore

@@ -0,0 +1,9 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/

data/.rspec

@@ -0,0 +1,2 @@
+--format documentation
+--color

data/.travis.yml

@@ -0,0 +1,4 @@
+language: ruby
+rvm:
+  - 2.2.3
+before_install: gem install bundler -v 1.10.6

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,13 @@
+# Contributor Code of Conduct
+
+As contributors and maintainers of this project, we pledge to respect all people who contribute through reporting issues, posting feature requests, updating documentation, submitting pull requests or patches, and other activities.
+
+We are committed to making participation in this project a harassment-free experience for everyone, regardless of level of experience, gender, gender identity and expression, sexual orientation, disability, personal appearance, body size, race, ethnicity, age, or religion.
+
+Examples of unacceptable behavior by participants include the use of sexual language or imagery, derogatory comments or personal attacks, trolling, public or private harassment, insults, or other unprofessional conduct.
+
+Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct. Project maintainers who do not follow the Code of Conduct may be removed from the project team.
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by opening an issue or contacting one or more of the project maintainers.
+
+This Code of Conduct is adapted from the [Contributor Covenant](http://contributor-covenant.org), version 1.0.0, available at [http://contributor-covenant.org/version/1/0/0/](http://contributor-covenant.org/version/1/0/0/)

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in retl.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2015 TODO: Write your name
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,336 @@
+# rETL
+
+**R**uby
+**E**xtract
+**T**ransform
+**L**oad
+
+rETL is a gem with a rich DSL for ETL (extract, transform, load) projects in 
+Ruby.
+
+## Transforming Data
+
+The core construct for transforming data is called a **Path**. A path describes
+how data should be transformed. rETL has a rich DSL that is intended to work on
+Hash objects, but it is not limited strictly to Hashes.
+
+### Defining a Path
+
+A path is composed of **Steps**. Each step is intended to act on a single unit
+of data that is passed into a block (like a row from a database query). The
+steps are executed in the order they are defined. In other words, the data
+passes through each step until it is completely transformed. In order to keep
+the Path easy to understand and maintain, each step should be responsible for a
+single action on a row.
+
+This path will `filter` data by `:first_name`, and then `transform` it by adding
+a `:full_name` key.
+
+```
+my_path = Retl::Path.new do
+  filter do |row|
+    row[:first_name] == "David"
+  end
+
+  transform do |row|
+    row[:full_name] = row[:first_name] + " " + row[:last_name]
+  end
+end
+```
+
+Once a path is defined, it can be used to transform data with the
+`#transform(enumerable, options={})` method. A `Transformation` object is
+returned, which is `Enumerable`.
+
+```
+data = [
+  { full_name: "David"  , last_name: "Biehl" },
+  { full_name: "Indiana", last_name: "Jones" }
+]
+
+result = my_path.transform(data)
+result.to_a
+#=> [ { full_name: "David", last_name: "Biehl", full_name: "David Biehl" } ]
+```
+
+### Available Steps
+
+There are several steps available in the rETL DSL that will transform data in
+different ways.
+
+#### Transform
+
+The `transform` step will mutate the data passed into the block. The return
+value of a `transform` step is ignored. See the `:full_name` example above.
+
+#### Replace
+
+The `replace` step will replace the data with the return value of the block. A
+common use is to use `replace` as the first step to convert incoming objects
+into hashes.
+
+```
+users = Users.where(active: true)  # a bunch of ActiveRecord objects
+
+my_path = Retl::Path.new do
+  replace do |user|
+    user.to_hash
+  end
+
+  # perform other steps with the hash
+end
+```
+
+#### Filter & Reject
+
+The `filter` step uses a predicate block to determine if the data should proceed
+to the next step. Conversely, `reject` will discard data if the predicate is
+truthy. `select` is an alias for `filter`.
+
+```
+data = [
+  {name: "David"  , age: 33}
+  {name: "Indiana", age: 50}
+  {name: "Sully"  , age: 7}
+  {name: "Boo"    , age: 3}
+]
+
+my_path = Retl::Path.new do 
+  transform do |row|
+    row[:adult_or_child] = row[:age] >= 18 ? "adult" : "child"
+  end
+
+  reject do |row| 
+    row[:adult_or_child] == "child" 
+  end
+
+  select do |row|    # `select` is an alias of `filter`
+    row[:age].odd?
+  end
+end
+
+result = my_path.transform(data)
+result.to_a
+#=> [ { name: "David", ... } ]
+```
+
+#### Calculate
+
+The `calculate` step will calculate a single key on a Hash with the return value
+of the block. For example, `calculate` can be used instead of `transform` from
+the first example. `calc` is a short-hand alias for `calculate`.
+
+```
+my_path = Retl::Path.new do
+  calculate(:full_name) do |row|
+    row[:first_name] + " " + row[:last_name]
+  end
+end
+```
+
+#### Inspect
+
+`inspect` steps cannot change the incoming data, and the return value of the
+block is ignored. These steps are intended for debugging, testing or logging
+purposes.
+
+```
+# rspec example
+
+it "adds a full name" do 
+  rspec = self
+
+  my_path = Retl::Path.new do
+    calc(:full_name) do |row|
+      row[:first_name] + " " + row[:last_name]
+    end
+  
+    inspect do |row|
+      rspec.expect(row).to rspec.include(full_name: "Indiana Jones")
+    end
+  end
+
+  my_path.transform([ { first_name: "Indiana", last_name: "Jones"} ]).to_a
+end
+```
+
+#### Fork
+
+Paths can be forked for alternate results with the `fork(name)` step. The forked
+data can then be accessed on the result with the `#forks(name)` method. Forks
+are unaffected by any steps that take place after the fork is defined.
+
+```
+data = [
+  { name: "David"  , age: 33 }
+  { name: "Indiana", age: 50 }
+  { name: "Sully"  , age: 7 }
+  { name: "Boo"    , age: 3 }
+]
+
+my_path = Retl::Path.new do 
+  transform do |row|
+    row[:adult_or_child] = row[:age] >= 18 ? "adult" : "child"
+  end
+
+  fork(:adults) do 
+    filter do |row|
+      row[:adult_or_child] == "adult"
+    end
+  end
+
+  fork(:children) do 
+    filter do |row|
+      row[:adult_or_child] == "child"
+    end
+  end
+
+  reject do |row| 
+    true  # oops, rejecting everything after the forks.  
+  end
+end
+
+result = my_path.transform(data)
+
+result.forks(:adults).to_a
+#=> [ { name: "David", ... }, { name: "Indiana", ... } ]
+
+result.forks(:children).to_a
+#=> [ { name: "Sully", ... }, { name: "Boo", ... } ]
+
+result.to_a
+#> [ ]
+```
+
+#### Explode
+
+The `explode` step adds additional data to the Path. The return value of the
+block should respond to `#each`, like an Array.
+
+```
+my_path = Reth::Path.new do
+  explode do |number|
+    number.times.map { |x| x + x + x }
+  end
+
+  filter do |number|
+    number.odd?
+  end
+end
+
+my_path.transform(6).to_a
+#=> [3, 9, 15]
+```
+
+#### Path Reuse
+
+In rETL, Paths can be re-used with the `step` step. Common Paths can be defined
+to ensure that calculations yield consistent results throughout the entire ETL
+project. Consistent data and meanings will make the data warehouse easier to
+understand for data consumers.
+
+```
+AdultOrChild = Retl::Path.new do 
+  calculate(:adult_or_child) do 
+    row[:age] >= 18 ? "adult" : "child"
+  end
+end
+
+my_path = Retl::Path.new do 
+  step AdultOrChild
+
+  ### perform other steps
+end
+```
+
+### Dependencies
+
+Dependencies can be defined with the `depends_on(name)`. The value of the
+dependency is accessible inside of each step by its name. In this example, we'll
+define an `age_lookup` dependency.
+
+```
+my_path = Retl::Path.new do
+  depends_on(:age_lookup) do 
+    { 
+      "adult" => "Adults are 18 or older",
+      "child" => "Children are younger than 18"
+    }
+  end
+
+  step AdultOrChild   # see previous example
+
+  calculate(:age_description) do |row|
+    age_lookup[row[:adult_or_child]]
+  end
+end
+```
+
+Dependencies can also be injected when the transformation takes place. This is
+useful for testing by passing in mocks or stubs. Also, concrete results from
+other paths can be merged merged into a single path making data integration
+possible.
+
+```
+my_path = Retl::Path.new do
+  depends_on(:age_lookup) do |options|  # transformation options are passed into `depends_on`
+    options[:age_lookup] || (raise ArgumentError, "This Path depends on an age lookup hash")
+  end
+
+  step AdultOrChild
+
+  calculate(:age_description) do |row|
+    age_lookup[row[:adult_or_child]]
+  end
+end
+
+age_lookup_hash = { 
+  "adult" => "Adults are 18 or older",
+  "child" => "Children are younger than 18"
+}
+
+my_path.transform(data, age_lookup: age_lookup_hash)
+```
+
+## Roadmap
+
+Currently the rETL gem's strengths are transforming data and code reuse. However
+this is only one part of an ETL project. I haven't even started on extracting or
+loading. Fortunately, the contract for transformation is very simple.
+
+```
+path.transform(Enumerable) 
+#=> Enumerable
+```
+
+Enumerales in, Enumerales out. This makes the application of the gem pretty much
+universal for any type of data transformation requirement in Ruby.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'retl'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install retl
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at
+https://github.com/davidbiehl/retl. This project is intended to be a safe,
+welcoming space for collaboration, and contributors are expected to adhere to
+the [Contributor Covenant](contributor-covenant.org) code of conduct.
+
+
+## License
+
+The gem is available as open source under the terms of the [MIT
+License](http://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,6 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task :default => :spec

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "retl"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start

data/bin/setup

@@ -0,0 +1,7 @@
+#!/bin/bash
+set -euo pipefail
+IFS=$'\n\t'
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/lib/retl.rb

@@ -0,0 +1,7 @@
+require "retl/version"
+
+require "retl/path"
+
+module Retl
+  # Your code goes here...
+end

data/lib/retl/context.rb

@@ -0,0 +1,29 @@
+require "retl/event_router"
+
+module Retl
+  class Context
+    def initialize(path, options={})
+      path.dependencies.each do |name, dependency|
+        self.class.send(:define_method, name) { dependency.call(options) }
+      end
+
+      @_events = EventRouter.new
+    end
+
+    def execute_step(step, data)
+      if step.is_a?(Proc)
+        instance_exec(data, self, &step)
+      else
+        if step.method(:call).arity.abs == 2
+          step.call(data, self)
+        else
+          step.call(data)
+        end
+      end
+    end
+
+    def _events
+      @_events
+    end
+  end
+end

data/lib/retl/event_router.rb

@@ -0,0 +1,22 @@
+module Retl
+  class EventRouter
+    def initialize
+      @listeners = {}
+    end
+
+    def listen_to(event_name, &block)
+      @listeners[event_name] ||= []
+      @listeners[event_name] << block
+    end
+
+    def trigger(event_name, args={})
+      listeners = @listeners[event_name]
+
+      if listeners
+        listeners.each do |handler|
+          handler.call(args)
+        end
+      end
+    end
+  end
+end

data/lib/retl/fork_data_collector.rb

@@ -0,0 +1,17 @@
+module Retl
+  class ForkDataCollector
+    def initialize(context)
+      @fork_data = {}
+
+      context._events.listen_to(:fork_data) do |args|
+        fork_name = args[:fork_name]
+        @fork_data[fork_name] ||= []
+        @fork_data[fork_name] << args[:data]
+      end
+    end
+
+    def take(name)
+      @fork_data.delete(name)
+    end
+  end
+end

data/lib/retl/handlers/explode_handler.rb

@@ -0,0 +1,11 @@
+require_relative "step_handler"
+
+module Retl
+  class ExplodeHandler < StepHandler
+    def push_in(data, context)
+      context.execute_step(step, data).each do |result|
+        push_out result
+      end
+    end
+  end
+end

data/lib/retl/handlers/filter_handler.rb

@@ -0,0 +1,10 @@
+require_relative "step_handler"
+
+module Retl
+  class FilterHandler < StepHandler
+    def push_in(data, context)
+      keep = context.execute_step(step, data)
+      push_out data if keep
+    end
+  end
+end

data/lib/retl/handlers/fork_handler.rb

@@ -0,0 +1,15 @@
+require_relative "handler"
+
+module Retl
+  class ForkHandler < Handler
+    def initialize(fork)
+      super()
+      @fork = fork
+    end
+
+    def push_in(data, context)
+      context._events.trigger(:fork_data, fork_name: @fork, data: data.dup)
+      push_out data
+    end
+  end
+end

data/lib/retl/handlers/handler.rb

@@ -0,0 +1,21 @@
+module Retl
+  class Handler
+    def initialize
+      @output = []
+    end
+
+    def output
+      @output.slice!(0, @output.count)
+    end
+
+    def push_in(data, context)
+      raise NotImplementedError, "Handlers much implement the #push_in(data, context) method"
+    end
+
+    private
+
+    def push_out(data)
+      @output.push data
+    end
+  end
+end

data/lib/retl/handlers/inspect_handler.rb

@@ -0,0 +1,10 @@
+require_relative "step_handler"
+
+module Retl
+  class InspectHandler < StepHandler
+    def push_in(data, context)
+      context.execute_step(step, data.dup)
+      push_out data
+    end
+  end
+end

data/lib/retl/handlers/step_handler.rb

@@ -0,0 +1,16 @@
+require_relative "handler"
+
+module Retl
+  class StepHandler < Handler
+    attr_reader :step
+
+    def initialize(step)
+      super()
+      @step = step
+    end
+
+    def push_in(data, context)
+      push_out context.execute_step(step, data)
+    end
+  end
+end

data/lib/retl/handlers/transform_handler.rb

@@ -0,0 +1,10 @@
+require_relative "step_handler"
+
+module Retl
+  class TransformHandler < StepHandler
+    def push_in(data, context)
+      context.execute_step(step, data)
+      push_out data
+    end
+  end
+end

data/lib/retl/path.rb

@@ -0,0 +1,178 @@
+require "retl/path_builder"
+require "retl/transformation"
+require "retl/context"
+require "retl/handlers/handler"
+require "retl/handlers/step_handler"
+require "retl/handlers/explode_handler"
+require "retl/handlers/fork_handler"
+
+module Retl
+  # A Path is a blueprint for transforming data
+  #
+  # A Path is a sequence of steps that are executed on data in order to 
+  # transform it.
+  # 
+  # Paths can be built with a block using the API defined in the {#PathBuilder}.
+  #
+  # Steps are added to the Path with the {#add_step} method.
+  #
+  # A Path can act on a single piece of data with the {#call} method.
+  #
+  # A Path can transform a list of data with the {#transform} method.
+  #
+  # @example
+  #   path = Retl::Path.new do 
+  #     step do |data|
+  #       data[:something] = "some value"
+  #       data
+  #     end
+  #
+  #     calculate(:something_else) do 
+  #       "some other value"
+  #     end
+  #     
+  #     transform do |data|
+  #       data[:something_together] = data[:something] + data[:something_else]
+  #     end
+  #   end
+  #
+  #   path.transform(data)
+  #
+  class Path
+    attr_reader :steps, :source, :dependencies
+
+    # Initializes a new Path
+    #
+    # @param parent [Path] - a Path to inherit from
+    def initialize(parent=nil, &block)
+      @steps         = []
+      @dependencies  = {}
+      @forks         = {}
+      @fork_builders = {}
+
+      if parent
+        @dependencies = parent.dependencies.dup
+        add_step parent.dup, handler: ExplodeHandler
+      end
+
+      build(&block) if block
+    end
+
+    # Builds a Path with the PathBuilder DSL
+    #
+    # @return [void]
+    def build(&block)
+      PathBuilder.new(self, &block)
+    end
+
+    # Initializer when copying a Path
+    #
+    # When a Path is copied, a copy of the Path's steps need to be copied
+    # as well. That was if additional steps are added to the original Path
+    # they won't be part of the copied Path.
+    def initialize_copy(source)
+      @steps         = source.steps.dup
+      @forks         = {}
+      @fork_builders = {}
+    end
+
+    # Adds a step to the Path
+    #
+    # A step is called with data and is expected to return complete, modified
+    # data.
+    #
+    # Steps are executed in the sequence they are added.
+    #
+    # @param step [#call(data)] the step to take
+    #
+    # @return [void]
+    def add_step(step, handler: StepHandler)
+      add_handler handler.new(step)
+    end
+
+    def add_handler(handler)
+      @steps << handler
+    end
+
+    # Execuutes the Path with the given data
+    #
+    # Currently the DSL mostly supports Hash based data, so this expects a 
+    # Hash.
+    #
+    # Since a piece of data can now be exploded, this method will always
+    # return an Array. 
+    #
+    # @param data    [Hash]    the data that will be transformed by the Path
+    # @param context [Context] the execution context for the transformation
+    #
+    # @return [Array<Hash>] the transformed data
+    def call(data, context=Context.new(self))
+      @steps.reduce([data]) do |queue, handler|
+        queue.each do |data|
+          handler.push_in(data, context)
+        end
+        handler.output
+      end
+    end
+
+    # Adds an fork to the Path
+    #
+    # Forks can be accessed via #forks
+    #
+    # @example
+    #   path.add_fork(:river) do 
+    #     filter { |data| data[:is_wet] }
+    #   end
+    #   path.forks(:river)
+    #
+    # @param name [Symbol] the name of the fork
+    #
+    # @return [void]
+    def add_fork(name, &block)
+      fork = Path.new(&block)
+      add_handler ForkHandler.new(name)
+      @forks[name] = fork 
+    end
+
+    # Gets a fork by name
+    #
+    # @param name [Symbol] the name of the fork to get
+    #
+    # @return [Path] the forked path
+    def forks(name)
+      @forks[name]
+    end
+
+    # Adds a fork builder block
+    #
+    # @param name   [Symbol] the name of the fork to build
+    # @param &block [Block]  the block that builds the fork
+    #
+    # @return [Fork] the built fork
+    def add_fork_builder(name, &block)
+      @fork_builders[name] = block
+      add_fork(name, &block)
+    end
+
+    # Adds a depdency to the Path
+    #
+    # @param name [Symbol] the name of the dependency
+    #   (should be a valid Ruby method)
+    # @param source [#call] a callable object that will return the depdency
+    #
+    # @return [void]
+    def add_dependency(name, source)
+      @dependencies[name] = source
+    end
+
+    # Executes the Path with data
+    #
+    # @param [Enumerable] the data that will be processed by the Path
+    # @option options that will be passed to #depends_on for the context
+    #
+    # @return [Transformation] the Tranformation of the data by the Path
+    def transform(enumerable, options={})
+      Transformation.new(enumerable, self, options)
+    end
+  end
+end

data/lib/retl/path_builder.rb

@@ -0,0 +1,62 @@
+require "retl/handlers/step_handler"
+require "retl/handlers/transform_handler"
+require "retl/handlers/filter_handler"
+require "retl/handlers/inspect_handler"
+require "retl/handlers/explode_handler"
+
+
+module Retl
+  class PathBuilder
+    def initialize(path, &block)
+      @path = path
+      instance_eval(&block)
+    end
+
+    def step(step=nil, handler: StepHandler, &block)
+      step ||= block
+      @path.add_step step, handler: handler
+    end
+    alias_method :replace, :step
+
+    def transform(action=nil, &block)
+      action ||= block
+      step(action, handler: TransformHandler)
+    end
+
+    def filter(predicate=nil, &block)
+      predicate ||= block
+      step(predicate, handler: FilterHandler)
+    end
+    alias_method :select, :filter
+
+    def fork(name, &block)
+      @path.add_fork_builder(name, &block)
+    end
+
+    def inspect(action=nil, &block)
+      action ||= block
+      step(action, handler: InspectHandler)
+    end
+
+    def reject(predicate=nil, &block)
+      predicate ||= block
+      filter { |data, context| !context.execute_step(predicate, data) }
+    end
+
+    def calculate(key, action=nil, &block)
+      action ||= block
+      transform { |data, context| data[key] = context.execute_step(action, data) }
+    end
+    alias_method :calc, :calculate
+
+    def depends_on(name, source=nil, &block)
+      source ||= block
+      @path.add_dependency(name, source)
+    end
+
+    def explode(action=nil, &block)
+      action ||= block
+      step(action, handler: ExplodeHandler)
+    end
+  end
+end

data/lib/retl/transformation.rb

@@ -0,0 +1,69 @@
+require "retl/context"
+require "retl/fork_data_collector"
+
+module Retl
+  class Transformation
+    include Enumerable
+    
+    def initialize(enumerable, path, options={})
+      @enumerable, @path, @options = enumerable, path, options
+      @context   = Context.new(@path, @options)
+      @fork_data = ForkDataCollector.new(@context)
+    end
+
+    def each(&block)
+      if @each
+        @each.each(&block)
+      else
+        build_each_result(&block)
+      end
+    end
+
+    def each_slice(size, &block)
+      @each_slice ||= {}
+      if @each_slice[size]
+        @each_slice[size].each(&block)
+      else
+        build_each_slice_result(size, &block)
+      end
+    end
+
+    def forks(name)
+      build_each_result
+      @path.forks(name).transform(@fork_data.take(name), @options)
+    end
+
+    def load_into(*destinations)
+      destinations = Array(destinations)
+
+      each do |data|
+        destinations.each do |destination|
+          destination << data
+        end
+      end
+
+      destinations.each do |destination|
+        destination.close if destination.respond_to?(:close)
+      end
+    end
+
+    private
+
+    def build_each_result(&block)
+      @each ||= @enumerable.reduce([]) do |result, data|
+        @path.call(data, @context).each do |data|
+          yield data if block_given?
+          result << data
+        end
+      end
+    end
+
+    def build_each_slice_result(size, &block)
+      @each_slice[size] ||= @enumerable.each_slice(size).reduce([]) do |result, slice|
+        transformed_slice = Transformation.new(slice, @path, @options)
+        yield transformed_slice if block_given?
+        result << transformed_slice
+      end
+    end
+  end
+end

data/lib/retl/version.rb

@@ -0,0 +1,3 @@
+module Retl
+  VERSION = "0.0.3"
+end

data/retl.gemspec

@@ -0,0 +1,25 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'retl/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "retl"
+  spec.version       = Retl::VERSION
+  spec.authors       = ["David Biehl"]
+  spec.email         = ["me@davidbiehl.com"]
+
+  spec.summary       = %q{Path based ETL processing in Ruby}
+  spec.description   = %q{Define paths for your data for ETL processing}
+  spec.homepage      = "https://github.com/davidbiehl/retl.git"
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+  spec.bindir        = "exe"
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+
+  spec.add_development_dependency "bundler", "~> 1.10"
+  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_development_dependency "rspec"
+end

metadata

@@ -0,0 +1,112 @@
+--- !ruby/object:Gem::Specification
+name: retl
+version: !ruby/object:Gem::Version
+  version: 0.0.3
+platform: ruby
+authors:
+- David Biehl
+autorequire: 
+bindir: exe
+cert_chain: []
+date: 2015-11-20 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.10'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.10'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: Define paths for your data for ETL processing
+email:
+- me@davidbiehl.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- ".rspec"
+- ".travis.yml"
+- CODE_OF_CONDUCT.md
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- bin/console
+- bin/setup
+- lib/retl.rb
+- lib/retl/context.rb
+- lib/retl/event_router.rb
+- lib/retl/fork_data_collector.rb
+- lib/retl/handlers/explode_handler.rb
+- lib/retl/handlers/filter_handler.rb
+- lib/retl/handlers/fork_handler.rb
+- lib/retl/handlers/handler.rb
+- lib/retl/handlers/inspect_handler.rb
+- lib/retl/handlers/step_handler.rb
+- lib/retl/handlers/transform_handler.rb
+- lib/retl/path.rb
+- lib/retl/path_builder.rb
+- lib/retl/transformation.rb
+- lib/retl/version.rb
+- retl.gemspec
+homepage: https://github.com/davidbiehl/retl.git
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.4.5.1
+signing_key: 
+specification_version: 4
+summary: Path based ETL processing in Ruby
+test_files: []