kiba 1.0.0 → 2.0.0.rc1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 622b10fe7f524f66152c10ab8618d6a661831776
-  data.tar.gz: 7410d79e2dec5bb48e9f1079e94c87260226d66c
+  metadata.gz: ca18859887a38d8eee1afa9ae170ad8fc2a8b63f
+  data.tar.gz: d27a6985d90a1ab73315b4f859bca2e105ba66a7
 SHA512:
-  metadata.gz: 5fe1c537d31ccc49446316f40c76630f724774431a3914019b870ce4cbdc3d34ed7eda4b66b622855d7d6972f435a0b8f831567f9b53b159e2836257b766671c
-  data.tar.gz: 60c2e8929bdd350308c5f377b0a6c989c3473f147903a90dab7ab9b4b58f6ec06cdc038094c5bca2ebd593defe2f0e7994752bdef325bd4ccbd7e04ef27f86a9
+  metadata.gz: dc66fedc5922ee6b63a4cfdd647e71592af3ae8d2851600008a5af0c165252583c1fdf15b064ff691cf19b6ebe004edcf1062fb0091c45f7b2c7e3a5e7e5bc27
+  data.tar.gz: d02d8e6293dbfa6ca2b0372ff36ab3548ef3608ce5554f44ee82a04f1d0a17cbb86838de5745721260a357943c11429012bbd3dbccd59abff08f5b29226f2454

data/.travis.yml

@@ -1,10 +1,15 @@
 language: ruby
 before_install:
+  # https://stackoverflow.com/a/47972768
+  - gem update --system
   - gem update bundler
 rvm:
-  - 2.3.0 # 2.3 won't work here (RVM issue afaik)
+  - ruby-head
+  - 2.5.0
+  - 2.4.3
+  - 2.3
   - 2.2
   - 2.1
   - 2.0
   - jruby-1.7
-  - jruby-9
+  - jruby-9.1.15.0

data/Changes.md

@@ -1,3 +1,9 @@
+2.0.0.rc1 (unreleased)
+----------------------
+
+- New (opt-in) StreamingRunner allows class transforms to generate more than one row. See [#44](https://github.com/thbar/kiba/pull/44) for rationale & how to activate.
+- Potentially breaking change if you were using the internal class `Kiba::Parser` directly: ETL jobs parsing has been modified to improve the isolation between the evaluation scope and the Kiba classes. See [#46](https://github.com/thbar/kiba/pull/46) for more information.
+
 1.0.0
 -----
 

data/LICENSE

@@ -0,0 +1,5 @@
+Copyright (c) LoGeek SARL
+
+Kiba Common is an Open Source project licensed under the terms of
+the LGPLv3 license.  Please see <http://www.gnu.org/licenses/lgpl-3.0.html>
+for license text.

data/Pro-Changes.md

@@ -0,0 +1,31 @@
+Kiba Pro Changelog
+==================
+
+Kiba Pro is the commercial extension for Kiba. Documentation is available on the [Wiki](https://github.com/thbar/kiba/wiki).
+
+HEAD
+-------
+
+1.0.0.rc1
+---------
+
+NOTE: documentation & requirements/compatibility are available on the [wiki](https://github.com/thbar/kiba/wiki).
+
+- New: `SQLUpsert` destination allowing row-by-row "insert or update".
+- New: `SQL` source allowing efficient streaming of large volumes of SQL rows while controlling memory consumption.
+- Improvement: `SQLBulkInsert` can now be used from a Sidekiq job.
+
+0.9.0
+-----
+
+- Multiple improvements to `SQLBulkInsert`:
+  - New flexible `row_pre_processor` option which allows to either remove a row conditionally (useful to conditionally target a given destination amongst many) or to replace it by N dynamically computed target rows.
+  - New callbacks: `after_initialize` & `before_flush` (useful to enforce dependent destinations flush & ensure required foreign keys constraints are respected).
+  - `logger` support.
+  - Bugfix: make sure to `disconnect` on `close`.
+  - Extra safety checks on row keys.
+
+0.4.0
+-----
+
+- Initial release of the `SQLBulkInsert` destination (providing fast SQL INSERT).

data/README.md

@@ -1,306 +1,59 @@
-**Foreword - if you need help**: please [ask your question with tag kiba-etl on StackOverflow](http://stackoverflow.com/questions/ask?tags=kiba-etl) so that other can benefit from your contribution! I monitor this specific tag and will reply to you.
+**If you need help**, please [ask your question with tag kiba-etl on StackOverflow](http://stackoverflow.com/questions/ask?tags=kiba-etl) so that other can benefit from your contribution! I monitor this specific tag and will reply to you.
 
 Writing reliable, concise, well-tested & maintainable data-processing code is tricky.
 
-Kiba lets you define and run such high-quality ETL ([Extract-Transform-Load](http://en.wikipedia.org/wiki/Extract,_transform,_load)) jobs, using Ruby (see [supported versions](#supported-ruby-versions)).
+Kiba lets you define and run such high-quality ETL ([Extract-Transform-Load](http://en.wikipedia.org/wiki/Extract,_transform,_load)) jobs using Ruby.
 
-Learn more on the [Kiba blog](http://thibautbarrere.com) and on [StackOverflow](http://stackoverflow.com/questions/tagged/kiba-etl):
-
-* [Live Coding Session - Processing data with Kiba ETL](http://thibautbarrere.com/2015/11/09/video-processing-data-with-kiba-etl/)
-* [Rubyists - are you doing ETL unknowningly?](http://thibautbarrere.com/2015/03/25/rubyists-are-you-doing-etl-unknowingly/)
-* [How to write solid data processing code](http://thibautbarrere.com/2015/04/05/how-to-write-solid-data-processing-code/)
-* [How to reformat CSV files with Kiba](http://thibautbarrere.com/2015/06/04/how-to-reformat-csv-files-with-kiba/) (in-depth, hands-on tutorial)
-* [How to explode multivalued attributes with Kiba ETL?](http://thibautbarrere.com/2015/06/25/how-to-explode-multivalued-attributes-with-kiba/)
-* [Common techniques to compute aggregates with Kiba](https://stackoverflow.com/questions/31145715/how-to-do-a-aggregation-transformation-in-a-kiba-etl-script-kiba-gem)
-* [How to run Kiba in a Rails environment?](http://thibautbarrere.com/2015/09/26/how-to-run-kiba-in-a-rails-environment/)
-
-**Consulting services**: if your organization needs to leverage data processing to solve a given business problem, I'm available to help you out via consulting sessions. [More information](http://thibautbarrere.com/hire-me/).
-
-**Kiba Pro**: I'm working on a Pro version ([read more here](https://github.com/thbar/kiba/issues/20)) which will provide more advanced features and built-in goodies in exchange for a yearly subscription. This will also make sure I can support Kiba for the many years to come. [Chime in](https://github.com/thbar/kiba/issues/20) if your company is interested!
+Learn more on the [Wiki](https://github.com/thbar/kiba/wiki), on my [blog](http://thibautbarrere.com) and on [StackOverflow](http://stackoverflow.com/questions/tagged/kiba-etl).
 
 [![Gem Version](https://badge.fury.io/rb/kiba.svg)](http://badge.fury.io/rb/kiba)
-[![Build Status](https://travis-ci.org/thbar/kiba.svg?branch=master)](https://travis-ci.org/thbar/kiba) [![Code Climate](https://codeclimate.com/github/thbar/kiba/badges/gpa.svg)](https://codeclimate.com/github/thbar/kiba) [![Dependency Status](https://gemnasium.com/thbar/kiba.svg)](https://gemnasium.com/thbar/kiba)
-
-## How do you define ETL jobs with Kiba?
-
-Kiba provides you with a DSL to define ETL jobs:
-
-```ruby
-# declare a ruby method here, for quick reusable logic
-def parse_french_date(date)
-  Date.strptime(date, '%d/%m/%Y')
-end
-
-# or better, include a ruby file which loads reusable assets
-# eg: commonly used sources / destinations / transforms, under unit-test
-require_relative 'common'
-
-# declare a pre-processor: a block called before the first row is read
-pre_process do
-  # do something
-end
-
-# declare a source where to take data from (you implement it - see notes below)
-source MyCsvSource, 'input.csv'
-
-# declare a row transform to process a given field
-transform do |row|
-  row[:birth_date] = parse_french_date(row[:birth_date])
-  # return to keep in the pipeline
-  row
-end
-
-# declare another row transform, dismissing rows conditionally by returning nil
-transform do |row|
-  row[:birth_date].year < 2000 ? row : nil
-end
-
-# declare a row transform as a class, which can be tested properly
-transform ComplianceCheckTransform, eula: 2015
-
-# before declaring a definition, maybe you'll want to retrieve credentials
-config = YAML.load(IO.read('config.yml'))
-
-# declare a destination - like source, you implement it (see below)
-destination MyDatabaseDestination, config['my_database']
-
-# declare a post-processor: a block called after all rows are successfully processed
-post_process do
-  # do something
-end
-```
-
-The combination of pre-processors, sources, transforms, destinations and post-processors defines the data processing pipeline.
-
-Note: you are advised to store your ETL definitions as files with the extension `.etl` (rather than `.rb`). This will make sure you do not end up loading them by mistake from another component (eg: a Rails app).
-
-## How do you run your ETL jobs?
-
-You can use the provided command-line:
-
-```
-bundle exec kiba my-data-processing-script.etl
-```
-
-This command essentially starts a two-step process:
-
-```ruby
-script_content = IO.read(filename)
-# pass the filename to get for line numbers on errors
-job_definition = Kiba.parse(script_content, filename)
-Kiba.run(job_definition)
-```
-
-`Kiba.parse` evaluates your ETL Ruby code to register sources, transforms, destinations and post-processors in a job definition. It is important to understand that you can use Ruby logic at the DSL parsing time. This means that such code is possible, provided the CSV files are available at parsing time:
-
-```ruby
-Dir['to_be_processed/*.csv'].each do |file|
-  source MyCsvSource, file
-end
-```
-
-Once the job definition is loaded, `Kiba.run` will use that information to do the actual row-by-row processing. It currently uses a simple row-by-row, single-threaded processing that will stop at the first error encountered.
-
-## Implementing ETL sources
-
-In Kiba, you are responsible for implementing the sources that do the extraction of data.
-
-Sources are classes implementing:
-- a constructor (to which Kiba will pass the provided arguments in the DSL)
-- the `each` method (which should yield rows one by one)
-
-Rows are usually `Hash` instances, but could be other structures as long as the rest of your pipeline is expecting it.
-
-Since sources are classes, you can (and are encouraged to) unit test them and reuse them.
-
-Here is a simple CSV source:
-
-```ruby
-require 'csv'
-
-class MyCsvSource
-  def initialize(input_file)
-    @csv = CSV.open(input_file, headers: true, header_converters: :symbol)
-  end
-
-  def each
-    @csv.each do |row|
-      yield(row.to_hash)
-    end
-    @csv.close
-  end
-end
-```
-
-## Implementing row transforms
-
-Row transforms can implemented in two ways: as blocks, or as classes.
-
-### Row transform as a block
+[![Build Status](https://travis-ci.org/thbar/kiba.svg?branch=master)](https://travis-ci.org/thbar/kiba) [![Build status](https://ci.appveyor.com/api/projects/status/v05jcyhpp1mueq9i?svg=true)](https://ci.appveyor.com/project/thbar/kiba) [![Code Climate](https://codeclimate.com/github/thbar/kiba/badges/gpa.svg)](https://codeclimate.com/github/thbar/kiba) [![Dependency Status](https://gemnasium.com/thbar/kiba.svg)](https://gemnasium.com/thbar/kiba)
 
-When writing a row transform as a block, it will be passed the row as parameter:
+## Note on upcoming Kiba 2.0.0
 
-```ruby
-transform do |row|
-  row[:this_field] = row[:that_field] * 10
-  # make sure to return the row to keep it in the pipeline
-  row
-end
-```
+Kiba 2.0.0 (available on `master`) includes an improved engine called the `StreamingRunner`, which allows transforms to generate more than one output row for each input row. See [#44](https://github.com/thbar/kiba/pull/44) for documentation on benefits & how to activate.
 
-To dismiss a row from the pipeline, simply return `nil` from a transform:
+## Getting Started
 
-```ruby
-transform { |row| row[:index] % 2 == 0 ? row : nil }
-```
+* [How do you define ETL jobs with Kiba?](https://github.com/thbar/kiba/wiki/How-do-you-define-ETL-jobs-with-Kiba%3F)
+* [How do you run your ETL jobs?](https://github.com/thbar/kiba/wiki/How-do-you-run-your-ETL-jobs%3F)
+* [Implementing ETL sources](https://github.com/thbar/kiba/wiki/Implementing-ETL-sources).
+* [Implementing ETL transforms](https://github.com/thbar/kiba/wiki/Implementing-ETL-transforms).
+* [Implementing ETL destinations](https://github.com/thbar/kiba/wiki/Implementing-ETL-destinations).
+* [Implementing pre and post-processors](https://github.com/thbar/kiba/wiki/Implementing-pre-and-post-processors).
 
-### Row transform as a class
+## Useful links
 
-If you implement the transform as a class, it must respond to `process(row)`:
-
-```ruby
-class SamplingTransform
-  def initialize(modulo_value)
-    @modulo_value = modulo_value
-  end
-
-  def process(row)
-    row[:index] % @modulo_value == 0 ? row : nil
-  end
-end
-```
-
-You'll use it this way in your ETL declaration (the parameters will be passed to initialize):
-
-```ruby
-# only keep 1 row over 10
-transform SamplingTransform, 10
-```
-
-Like the block form, it can return `nil` to dismiss the row. The class form allows better testability and reusability across your(s) ETL script(s).
-
-## Implementing ETL destinations
-
-Like sources, destinations are classes that you are providing. Destinations must implement:
-- a constructor (to which Kiba will pass the provided arguments in the DSL)
-- a `write(row)` method that will be called for each non-dismissed row
-- an optional `close` method that will be called, if present, at the end of the processing (useful to tear down resources such as connections)
-
-Here is an example destination:
-
-```ruby
-require 'csv'
-
-# simple destination assuming all rows have the same fields
-class MyCsvDestination
-  def initialize(output_file)
-    @csv = CSV.open(output_file, 'w')
-  end
-
-  def write(row)
-    unless @headers_written
-      @headers_written = true
-      @csv << row.keys
-    end
-    @csv << row.values
-  end
-
-  def close
-    @csv.close
-  end
-end
-```
-
-## Implementing pre and post-processors
-
-Pre-processors and post-processors are currently blocks, which get called only once per ETL run:
-- Pre-processors get called before the ETL starts reading rows from the sources.
-- Post-processors get invoked after the ETL successfully processed all the rows.
-
-Note that post-processors won't get called if an error occurred earlier.
-
-```ruby
-count = 0
-
-def system!(cmd)
-  fail "Command #{cmd} failed" unless system(cmd)
-end
-
-file = 'my_file.csv'
-sample_file = 'my_file.sample.csv'
-
-pre_process do
-  # it's handy to work with a reduced data set. you can
-  # e.g. just keep one line of the CSV files + the headers
-  system! "sed -n \"1p;25706p\" #{file} > #{sample_file}"
-end
-
-source MyCsv, file: sample_file
-
-transform do |row|
-  count += 1
-  row
-end
-
-post_process do
-  Email.send(supervisor_address, "#{count} rows successfully processed")
-end
-```
-
-## Composability, reusability, testability of Kiba components
-
-The way Kiba works makes it easy to create reusable, well-tested ETL components and jobs.
-
-The main reason for this is that a Kiba ETL script can `require` shared Ruby code, which allows to:
-- create well-tested, reusable sources & destinations
-- create macro-transforms as methods, to be reused across sister scripts
-- substitute a component by another (e.g.: try a variant of a destination)
-- use a centralized place for configuration (credentials, IP addresses, etc.)
-
-The fact that the DSL evaluation "runs" the script also allows for simple meta-programming techniques, like pre-reading a source file to extract field names, to be used in transform definitions.
-
-The ability to support that DSL, but also check command line arguments, environment variables and tweak behaviour as needed, or call other/faster specialized tools make Ruby an asset to implement ETL jobs.
-
-Make sure to subscribe to my [Ruby ETL blog](http://thibautbarrere.com) where I'll demonstrate such techniques over time!
+* [Live Coding Session - Processing data with Kiba ETL](http://thibautbarrere.com/2015/11/09/video-processing-data-with-kiba-etl/)
+* [Rubyists - are you doing ETL unknowningly?](http://thibautbarrere.com/2015/03/25/rubyists-are-you-doing-etl-unknowingly/)
+* [How to write solid data processing code](http://thibautbarrere.com/2015/04/05/how-to-write-solid-data-processing-code/)
+* [How to reformat CSV files with Kiba](http://thibautbarrere.com/2015/06/04/how-to-reformat-csv-files-with-kiba/) (in-depth, hands-on tutorial)
+* [How to explode multivalued attributes with Kiba ETL?](http://thibautbarrere.com/2015/06/25/how-to-explode-multivalued-attributes-with-kiba/)
+* [Common techniques to compute aggregates with Kiba](https://stackoverflow.com/questions/31145715/how-to-do-a-aggregation-transformation-in-a-kiba-etl-script-kiba-gem)
+* [How to run Kiba in a Rails environment?](http://thibautbarrere.com/2015/09/26/how-to-run-kiba-in-a-rails-environment/)
+* [How to pass parameters to the Kiba command line?](http://stackoverflow.com/questions/32959692/how-to-pass-parameters-into-your-etl-job)
 
 ## Supported Ruby versions
 
 Kiba currently supports Ruby 2.0+ and JRuby (with its default 1.9 syntax). See [test matrix](https://travis-ci.org/thbar/kiba).
 
-## History & Credits
-
-Wow, you're still there? Nice to meet you. I'm [Thibaut](http://thibautbarrere.com), author of Kiba.
-
-I first met the idea of row-based syntax when I started using [Anthony Eden](https://github.com/aeden)'s [Activewarehouse-ETL](https://github.com/activewarehouse/activewarehouse-etl), first published around 2006 (I think), in which Anthony applied the core principles defined by Ralph Kimball in [The Data Warehouse ETL Toolkit](http://www.amazon.com/gp/product/0764567578).
-
-I've been writing and maintaining a number of production ETL systems using Activewarehouse-ETL, then later with an ancestor of Kiba which was named TinyTL.
+## Kiba Common
 
-I took over the maintenance of Activewarehouse-ETL circa 2009/2010, but over time, I could not properly update & document it, given the gradual failure of a large number of dependencies and components. Ultimately in 2014 I had to stop maintaining it, after an already long hiatus.
+I'm starting to add commonly used reusable helpers in a separate gem called [kiba-common](https://github.com/thbar/kiba-common), check it out (work-in-progress).
 
-That said using Activewarehouse-ETL for so long made me realize the row-based processing syntax was great and provided some great assets for maintainability on long time-spans.
+## ETL consulting & commercial version
 
-Kiba is a completely fresh & minimalistic-on-purpose implementation of that row-based processing pattern.
+**Consulting services**: if your organization needs help to implement a data pipeline or to build a data-intensive application, I provide consulting services. [More information](http://thibautbarrere.com/hire-me/).
 
-It is minimalistic to make it more likely that I will be able to maintain it over time.
-
-It makes strong simplicity assumptions (like letting you define the sources, transforms & destinations). MiniTest is an inspiration.
-
-As I developed Kiba, I realize how much this simplicity opens the road for interesting developments such as multi-threaded & multi-processes processing.
-
-Last word: Kiba is 100% sponsored by my company LoGeek SARL (also provider of [WiseCash, a lightweight cash-flow forecasting app](https://www.wisecashhq.com)).
+**Kiba Pro**: for more features & goodies, check out Kiba Pro ([Changelog & contact info](Pro-Changes.md)).
 
 ## License
 
-Copyright (c) LoGeek SARL.
-
-Kiba is an Open Source project licensed under the terms of
-the LGPLv3 license.  Please see <http://www.gnu.org/licenses/lgpl-3.0.html>
-for license text.
+Copyright (c) LoGeek SARL. Kiba is an Open Source project licensed under the terms of
+the LGPLv3 license.  Please see <http://www.gnu.org/licenses/lgpl-3.0.html> for license text.
 
 ## Contributing & Legal
 
-Until the API is more stable, I can only accept documentation Pull Requests.
-
 (agreement below borrowed from [Sidekiq Legal](https://github.com/mperham/sidekiq/blob/master/Contributing.md))
 
 By submitting a Pull Request, you disavow any rights or claims to any changes submitted to the Kiba project and assign the copyright of those changes to LoGeek SARL.

data/appveyor.yml

@@ -1,18 +1,26 @@
-version: '{build}'
+version: 1.0.{build}-{branch}
 
-skip_tags: true
+cache:
+  - vendor/bundle
 
 environment:
   matrix:
-    - ruby_version: "21"
-    - ruby_version: "21-x64"
+    - RUBY_VERSION: 24
+    - RUBY_VERSION: 23
+    - RUBY_VERSION: 22
+    - RUBY_VERSION: 21
 
 install:
-  - SET PATH=C:\Ruby%ruby_version%\bin;%PATH%
-  - gem install bundler --no-document -v 1.10.5
-  - bundle install --retry=3
+  - set PATH=C:\Ruby%RUBY_VERSION%\bin;%PATH%
+  - bundle config --local path vendor/bundle
+  - bundle install
+
+build: off
+
+before_test:
+  - ruby -v
+  - gem -v
+  - bundle -v
 
 test_script:
   - bundle exec rake
-
-build: off

data/kiba.gemspec

@@ -15,6 +15,7 @@ Gem::Specification.new do |gem|
   gem.executables   = ['kiba']
 
   gem.add_development_dependency 'rake'
-  gem.add_development_dependency 'minitest'
+  gem.add_development_dependency 'minitest', '~> 5.9'
   gem.add_development_dependency 'awesome_print'
+  gem.add_development_dependency 'minitest-focus'
 end

data/lib/kiba.rb

@@ -5,6 +5,15 @@ require 'kiba/control'
 require 'kiba/context'
 require 'kiba/parser'
 require 'kiba/runner'
+require 'kiba/streaming_runner'
+require 'kiba/dsl_extensions/config'
 
 Kiba.extend(Kiba::Parser)
-Kiba.extend(Kiba::Runner)
+
+module Kiba
+  def self.run(job)
+    # NOTE: use Hash#dig when Ruby 2.2 reaches EOL
+    runner = job.config.fetch(:kiba, {}).fetch(:runner, Kiba::Runner)
+    runner.run(job)
+  end
+end

data/lib/kiba/control.rb

@@ -3,6 +3,10 @@ module Kiba
     def pre_processes
       @pre_processes ||= []
     end
+    
+    def config
+      @config ||= {}
+    end
 
     def sources
       @sources ||= []

data/lib/kiba/dsl_extensions/config.rb

@@ -0,0 +1,9 @@
+module Kiba
+  module DSLExtensions
+    module Config
+      def config(context, context_config)
+        (@control.config[context] ||= {}).merge!(context_config)
+      end
+    end
+  end
+end

data/lib/kiba/parser.rb

@@ -1,15 +1,26 @@
-module Kiba
-  module Parser
-    def parse(source_as_string = nil, source_file = nil, &source_as_block)
-      control = Control.new
-      context = Context.new(control)
-      if source_as_string
-        # this somewhat weird construct allows to remove a nil source_file
-        context.instance_eval(*[source_as_string, source_file].compact)
-      else
-        context.instance_eval(&source_as_block)
-      end
-      control
+# NOTE: using the "Kiba::Parser" declaration, as I discovered,
+# provides increased isolation to the declared ETL script, compared
+# to 2 nested modules.
+# Before that, a user creating entities named Control, Context
+# or DSLExtensions would see a conflict with Kiba own classes,
+# as by default instance_eval will resolve references by adding
+# the module containing the parser class (initially "Kiba").
+# Now, the classes appear to be further hidden from the user,
+# as Kiba::Parser is its own module.
+# This allows the user to create a Parser, Context, Control class
+# without it being interpreted as reopening Kiba::Parser, Kiba::Context,
+# etc.
+# See test in test_cli.rb (test_namespace_conflict)
+module Kiba::Parser
+  def parse(source_as_string = nil, source_file = nil, &source_as_block)
+    control = Kiba::Control.new
+    context = Kiba::Context.new(control)
+    if source_as_string
+      # this somewhat weird construct allows to remove a nil source_file
+      context.instance_eval(*[source_as_string, source_file].compact)
+    else
+      context.instance_eval(&source_as_block)
     end
+    control
   end
 end

data/lib/kiba/runner.rb

@@ -1,5 +1,7 @@
 module Kiba
   module Runner
+    extend self
+
     # allow to handle a block form just like a regular transform
     class AliasingProc < Proc
       alias_method :process, :call
@@ -13,8 +15,9 @@ module Kiba
       process_rows(
         to_instances(control.sources),
         to_instances(control.transforms, true),
-        to_instances(control.destinations)
+        destinations = to_instances(control.destinations)
       )
+      close_destinations(destinations)
       # TODO: when I add post processes as class, I'll have to add a test to
       # make sure instantiation occurs after the main processing is done (#16)
       run_post_processes(control)
@@ -28,6 +31,12 @@ module Kiba
       to_instances(control.post_processes, true, false).each(&:call)
     end
 
+    def close_destinations(destinations)
+      destinations
+      .find_all { |d| d.respond_to?(:close) }
+      .each(&:close)
+    end
+
     def process_rows(sources, transforms, destinations)
       sources.each do |source|
         source.each do |row|
@@ -41,7 +50,6 @@ module Kiba
           end
         end
       end
-      destinations.find_all { |d| d.respond_to?(:close) }.each(&:close)
     end
 
     # not using keyword args because JRuby defaults to 1.9 syntax currently

data/lib/kiba/streaming_runner.rb

@@ -0,0 +1,33 @@
+module Kiba
+  module StreamingRunner
+    include Runner
+    extend self
+    
+    def transform_stream(stream, t)
+      Enumerator.new do |y|
+        stream.each do |input_row|
+          returned_row = t.process(input_row) do |yielded_row|
+            y << yielded_row
+          end
+          y << returned_row if returned_row
+        end
+      end
+    end
+    
+    def source_stream(sources)
+      Enumerator.new do |y|
+        sources.each do |source|
+          source.each { |r| y << r }
+        end
+      end
+    end
+
+    def process_rows(sources, transforms, destinations)
+      stream = source_stream(sources)
+      recurser = lambda { |s,t| transform_stream(s, t) }
+      transforms.inject(stream, &recurser).each do |r|
+        destinations.each { |d| d.write(r) }
+      end
+    end
+  end
+end

data/lib/kiba/version.rb

@@ -1,3 +1,3 @@
 module Kiba
-  VERSION = '1.0.0'
+  VERSION = '2.0.0.rc1'
 end

data/test/common/runner.rb

@@ -0,0 +1,137 @@
+require 'minitest/mock'
+require_relative '../support/test_enumerable_source'
+
+module SharedRunnerTests
+  def kiba_run(job)
+    Kiba.run(job)
+  end
+
+  def rows
+    @rows ||= [
+      { identifier: 'first-row' },
+      { identifier: 'second-row' }
+    ]
+  end
+  
+  def control
+    @control ||= begin
+      control = Kiba::Control.new
+      # this will yield a single row for testing
+      control.sources << {
+        klass: TestEnumerableSource,
+        args: [rows]
+      }
+      control
+    end
+  end
+  
+  def test_block_transform_processing
+    # is there a better way to assert a block was called in minitest?
+    control.transforms << { block: lambda { |r| @called = true; r } }
+    kiba_run(control)
+    assert_equal true, @called
+  end
+
+  def test_dismissed_row_not_passed_to_next_transform
+    @called = nil
+    control.transforms << { block: lambda { |_| nil } }
+    control.transforms << { block: lambda { |_| @called = true; nil } }
+    kiba_run(control)
+    assert_nil @called
+  end
+
+  def test_post_process_runs_once
+    assert_equal 2, rows.size
+    @called = 0
+    control.post_processes << { block: lambda { @called += 1 } }
+    kiba_run(control)
+    assert_equal 1, @called
+  end
+
+  def test_post_process_not_called_after_row_failure
+    @called = nil
+    control.transforms << { block: lambda { |_| fail 'FAIL' } }
+    control.post_processes << { block: lambda { @called = true } }
+    assert_raises(RuntimeError, 'FAIL') { kiba_run(control) }
+    assert_nil @called
+  end
+  
+  def test_pre_process_runs_once
+    assert_equal 2, rows.size
+    @called = 0
+    control.pre_processes << { block: lambda { @called += 1 } }
+    kiba_run(control)
+    assert_equal 1, @called
+  end
+
+  def test_pre_process_runs_before_source_is_instantiated
+    calls = []
+
+    mock_source_class = MiniTest::Mock.new
+    mock_source_class.expect(:new, TestEnumerableSource.new([1, 2, 3])) do
+      calls << :source_instantiated
+    end
+
+    control = Kiba::Control.new
+    control.pre_processes << { block: lambda { calls << :pre_processor_executed } }
+    control.sources << { klass: mock_source_class }
+    kiba_run(control)
+
+    assert_equal [:pre_processor_executed, :source_instantiated], calls
+    assert_mock mock_source_class
+  end
+  
+  def test_no_error_raised_if_destination_close_not_implemented
+    # NOTE: this fake destination does not implement `close`
+    destination_instance = MiniTest::Mock.new
+
+    mock_destination_class = MiniTest::Mock.new
+    mock_destination_class.expect(:new, destination_instance)
+    
+    control = Kiba::Control.new
+    control.destinations << { klass: mock_destination_class }
+    kiba_run(control)
+    assert_mock mock_destination_class
+  end
+  
+  def test_destination_close_called_if_defined
+    destination_instance = MiniTest::Mock.new
+    destination_instance.expect(:close, nil)
+    mock_destination_class = MiniTest::Mock.new
+    mock_destination_class.expect(:new, destination_instance)
+
+    control = Kiba::Control.new
+    control.destinations << { klass: mock_destination_class }
+    kiba_run(control)
+    assert_mock destination_instance
+    assert_mock mock_destination_class
+  end
+  
+  def test_use_next_to_exit_early_from_block_transform
+    assert_equal 2, rows.size
+
+    # calling "return row" from a block is forbidden, but you can use "next" instead
+    b = lambda do |row|
+      if row.fetch(:identifier) == 'first-row'
+        # demonstrate how to remove a row from the pipeline via next
+        next
+      else
+        # demonstrate how you can reformat via next
+        next({new_identifier: row.fetch(:identifier)})
+      end
+      fail "This should not be called"
+    end
+    control.transforms << { block: b }
+
+    # keep track of the rows
+    @remaining_rows = []
+    checker = lambda { |row| @remaining_rows << row; row }
+    control.transforms << { block: checker }
+
+    kiba_run(control)
+    
+    # the first row should have been removed
+    # and the second row should have been reformatted
+    assert_equal [{new_identifier: 'second-row'}], @remaining_rows
+  end
+end

data/test/fixtures/namespace_conflict.etl

@@ -0,0 +1,9 @@
+fail "Context should not be visible without Kiba namespace" if defined?(Context)
+fail "Control should not be visible without Kiba namespace" if defined?(Control)
+fail "Parser should not be visible without Kiba namespace" if defined?(Parser)
+fail "Config should not be visible without Kiba namespace" if defined?(DSLExtensions::Config)
+
+# verify Kiba config (namespaced under Kiba::DSLExtensions::Config)
+# isn't causing troubles to implementers using a top-level DSLExtensions module
+require_relative 'some_extension'
+extend DSLExtensions::SomeExtension

data/test/fixtures/some_extension.rb

@@ -0,0 +1,4 @@
+module DSLExtensions
+  module SomeExtension
+  end
+end

data/test/helper.rb

@@ -1,5 +1,6 @@
 require 'minitest/autorun'
 require 'minitest/pride'
+require 'minitest/focus'
 require 'kiba'
 
 class Kiba::Test < Minitest::Test

data/test/support/shared_tests.rb

@@ -0,0 +1,10 @@
+module SharedTests
+  def shared_tests_for(desc, &block)
+    @@shared_tests ||= {}
+    @@shared_tests[desc] = block
+  end
+
+  def shared_tests(desc, *args)
+    self.class_exec(*args, &@@shared_tests.fetch(desc))
+  end
+end

data/test/support/test_array_destination.rb

@@ -0,0 +1,9 @@
+class TestArrayDestination
+  def initialize(array)
+    @array = array
+  end
+  
+  def write(row)
+    @array << row
+  end
+end

data/test/support/test_yielding_transform.rb

@@ -0,0 +1,8 @@
+class TestYieldingTransform
+  def process(row)
+    row.fetch(:tags).each do |value|
+      yield({item: value})
+    end
+    {item: "classic-return-value"}
+  end
+end

data/test/test_cli.rb

@@ -14,4 +14,8 @@ class TestCli < Kiba::Test
     assert_match(/uninitialized constant(.*)UnknownThing/, exception.message)
     assert_includes exception.backtrace.to_s, 'test/fixtures/bogus.etl:2:in'
   end
+
+  def test_namespace_conflict
+    Kiba::Cli.run([fixture('namespace_conflict.etl')])
+  end
 end

data/test/test_parser.rb

@@ -14,6 +14,17 @@ class TestParser < Kiba::Test
     assert_equal DummyClass, control.sources[0][:klass]
     assert_equal %w(has args), control.sources[0][:args]
   end
+  
+  # NOTE: useful for anything not using the CLI (e.g. sidekiq)
+  def test_block_parsing_with_reference_to_outside_variable
+    some_variable = Object.new
+    
+    control = Kiba.parse do
+      source DummyClass, some_variable
+    end
+    
+    assert_equal [some_variable], control.sources[0][:args]
+  end
 
   def test_block_transform_definition
     control = Kiba.parse do
@@ -89,4 +100,31 @@ RUBY
   ensure
     remove_files('test/tmp/etl-common.rb', 'test/tmp/etl-main.rb')
   end
+  
+  def test_config
+    control = Kiba.parse do
+      extend Kiba::DSLExtensions::Config
+      
+      config :context, key: "value", other_key: "other_value"
+    end
+    
+    assert_equal({ context: {
+      key: "value",
+      other_key: "other_value"
+    }}, control.config)
+  end
+  
+  def test_config_override
+    control = Kiba.parse do
+      extend Kiba::DSLExtensions::Config
+
+      config :context, key: "value", other_key: "other_value"
+      config :context, key: "new_value"
+    end
+    
+    assert_equal({ context: {
+      key: "new_value",
+      other_key: "other_value"
+    }}, control.config)
+  end
 end

data/test/test_runner.rb

@@ -1,87 +1,6 @@
 require_relative 'helper'
-require 'minitest/mock'
-require_relative 'support/test_enumerable_source'
+require_relative 'common/runner'
 
 class TestRunner < Kiba::Test
-  let(:rows) do
-    [
-      { field: 'value' },
-      { field: 'other-value' }
-    ]
-  end
-  
-  let(:control) do
-    control = Kiba::Control.new
-    # this will yield a single row for testing
-    control.sources << {
-      klass: TestEnumerableSource,
-      args: [rows]
-    }
-    control
-  end
-
-  def test_block_transform_processing
-    # is there a better way to assert a block was called in minitest?
-    control.transforms << { block: lambda { |r| @called = true; r } }
-    Kiba.run(control)
-    assert_equal true, @called
-  end
-
-  def test_dismissed_row_not_passed_to_next_transform
-    control.transforms << { block: lambda { |_| nil } }
-    control.transforms << { block: lambda { |_| @called = true; nil } }
-    Kiba.run(control)
-    assert_nil @called
-  end
-
-  def test_post_process_runs_once
-    assert_equal 2, rows.size
-    @called = 0
-    control.post_processes << { block: lambda { @called += 1 } }
-    Kiba.run(control)
-    assert_equal 1, @called
-  end
-
-  def test_post_process_not_called_after_row_failure
-    control.transforms << { block: lambda { |_| fail 'FAIL' } }
-    control.post_processes << { block: lambda { @called = true } }
-    assert_raises(RuntimeError, 'FAIL') { Kiba.run(control) }
-    assert_nil @called
-  end
-  
-  def test_pre_process_runs_once
-    assert_equal 2, rows.size
-    @called = 0
-    control.pre_processes << { block: lambda { @called += 1 } }
-    Kiba.run(control)
-    assert_equal 1, @called
-  end
-
-  def test_pre_process_runs_before_source_is_instantiated
-    calls = []
-
-    mock_source_class = MiniTest::Mock.new
-    mock_source_class.expect(:new, TestEnumerableSource.new([1, 2, 3])) do
-      calls << :source_instantiated
-    end
-
-    control = Kiba::Control.new
-    control.pre_processes << { block: lambda { calls << :pre_processor_executed } }
-    control.sources << { klass: mock_source_class }
-    Kiba.run(control)
-
-    assert_equal [:pre_processor_executed, :source_instantiated], calls
-  end
-  
-  def test_no_error_raised_if_destination_close_not_implemented
-    # NOTE: this fake destination does not implement `close`
-    destination_instance = MiniTest::Mock.new
-
-    mock_destination_class = MiniTest::Mock.new
-    mock_destination_class.expect(:new, destination_instance)
-    
-    control = Kiba::Control.new
-    control.destinations << { klass: mock_destination_class }
-    Kiba.run(control)
-  end
+  include SharedRunnerTests
 end

data/test/test_streaming_runner.rb

@@ -0,0 +1,33 @@
+require_relative 'helper'
+require_relative 'support/test_enumerable_source'
+require_relative 'support/test_array_destination'
+require_relative 'support/test_yielding_transform'
+require_relative 'common/runner'
+
+class TestStreamingRunner < Kiba::Test
+  include SharedRunnerTests
+  
+  def test_yielding_class_transform
+    input_row = {tags: ["one", "two", "three"]}
+    destination_array = []
+    
+    job = Kiba.parse do
+      extend Kiba::DSLExtensions::Config
+
+      config :kiba, runner: Kiba::StreamingRunner
+
+      source TestEnumerableSource, [input_row]
+      transform TestYieldingTransform
+      destination TestArrayDestination, destination_array
+    end
+    
+    kiba_run(job)
+  
+    assert_equal [
+      {item: 'one'},
+      {item: 'two'},
+      {item: 'three'},
+      {item: 'classic-return-value'}
+    ], destination_array
+  end
+end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: kiba
 version: !ruby/object:Gem::Version
-  version: 1.0.0
+  version: 2.0.0.rc1
 platform: ruby
 authors:
 - Thibaut Barrère
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2016-12-01 00:00:00.000000000 Z
+date: 2017-12-27 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rake
@@ -26,6 +26,20 @@ dependencies:
         version: '0'
 - !ruby/object:Gem::Dependency
   name: minitest
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '5.9'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '5.9'
+- !ruby/object:Gem::Dependency
+  name: awesome_print
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
@@ -39,7 +53,7 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
-  name: awesome_print
+  name: minitest-focus
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
@@ -64,6 +78,8 @@ files:
 - ".travis.yml"
 - Changes.md
 - Gemfile
+- LICENSE
+- Pro-Changes.md
 - README.md
 - Rakefile
 - appveyor.yml
@@ -73,21 +89,30 @@ files:
 - lib/kiba/cli.rb
 - lib/kiba/context.rb
 - lib/kiba/control.rb
+- lib/kiba/dsl_extensions/config.rb
 - lib/kiba/parser.rb
 - lib/kiba/runner.rb
+- lib/kiba/streaming_runner.rb
 - lib/kiba/version.rb
+- test/common/runner.rb
 - test/fixtures/bogus.etl
+- test/fixtures/namespace_conflict.etl
+- test/fixtures/some_extension.rb
 - test/fixtures/valid.etl
 - test/helper.rb
+- test/support/shared_tests.rb
+- test/support/test_array_destination.rb
 - test/support/test_csv_destination.rb
 - test/support/test_csv_source.rb
 - test/support/test_enumerable_source.rb
 - test/support/test_rename_field_transform.rb
 - test/support/test_source_that_reads_at_instantiation_time.rb
+- test/support/test_yielding_transform.rb
 - test/test_cli.rb
 - test/test_integration.rb
 - test/test_parser.rb
 - test/test_runner.rb
+- test/test_streaming_runner.rb
 - test/tmp/.gitkeep
 homepage: http://thbar.github.io/kiba/
 licenses:
@@ -104,26 +129,33 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ">="
+  - - ">"
     - !ruby/object:Gem::Version
-      version: '0'
+      version: 1.3.1
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.4.8
+rubygems_version: 2.6.14
 signing_key: 
 specification_version: 4
 summary: Lightweight ETL for Ruby
 test_files:
+- test/common/runner.rb
 - test/fixtures/bogus.etl
+- test/fixtures/namespace_conflict.etl
+- test/fixtures/some_extension.rb
 - test/fixtures/valid.etl
 - test/helper.rb
+- test/support/shared_tests.rb
+- test/support/test_array_destination.rb
 - test/support/test_csv_destination.rb
 - test/support/test_csv_source.rb
 - test/support/test_enumerable_source.rb
 - test/support/test_rename_field_transform.rb
 - test/support/test_source_that_reads_at_instantiation_time.rb
+- test/support/test_yielding_transform.rb
 - test/test_cli.rb
 - test/test_integration.rb
 - test/test_parser.rb
 - test/test_runner.rb
+- test/test_streaming_runner.rb
 - test/tmp/.gitkeep