kiba 2.0.0.rc1 → 3.6.0

data/Pro-Changes.md

@@ -1,13 +1,90 @@
 Kiba Pro Changelog
 ==================
 
-Kiba Pro is the commercial extension for Kiba. Documentation is available on the [Wiki](https://github.com/thbar/kiba/wiki).
+Kiba Pro provides vendor-supported ETL extensions for Kiba. Your subscription funds the Open-Source development, thanks for considering it!
 
-HEAD
--------
+Learn more on the [Kiba website](https://www.kiba-etl.org/kiba-pro).
 
-1.0.0.rc1
----------
+Documentation is available on the [Wiki](https://github.com/thbar/kiba/wiki#kiba-pro).
+
+2.0.0
+-----
+
+- New: `SQLBulkLookup` transform allows to efficiently lookup values in SQL tables. This is particularly useful in datawarehouse scenarios (to replace unique business keys by surrogate keys), or when writing migrations of SQL databases. Instead of looking-up each row individually, it avoids a "N+1" like effect, by working on large batches of rows.
+- New: `ParallelTransform` provides an easy way to process a group of ETL rows at the same time using a pool of threads. It can be used to accelerate ETL transforms doing IO operations such as HTTP queries, by going multithreaded.
+- New: `FileLock` adds an easy way to avoid overlapping runs in ETL Jobs using a local file lock.
+
+1.5.0
+-----
+
+- Compatibility with Kiba v3
+- BREAKING CHANGE: deprecate non-live Sequel connection passing (https://github.com/thbar/kiba/issues/79). Do not use `database: "connection_string"`, instead pass your `Sequel` connection directly. This moves the connection management out of the destination, which is a better pattern & provides better (block-based) resources closing.
+- Official MySQL support:
+  - While the compatibility was already here, it is now tested for in our QA testing suite.
+  - MySQL 5.5-8.0 is supported & tested
+  - MariaDB should be supported (although not tested against in the QA testing suite)
+  - Amazon Aurora MySQL is also supposed to work (although not tested)
+  - `Kiba::Pro::Sources::SQL` supports for non-streaming + streaming use
+  - `Kiba::Pro::Destinations::SQLBulkInsert` supports:
+    - Bulk insert
+    - Bulk insert with ignore
+    - Bulk upsert (including with dynamically computed columns) via `ON DUPLICATE KEY UPDATE`
+  - Note that the `Kiba::Pro::Destinations::SQLUpsert` (row-by-row) is not MySQL compatible at the moment
+
+1.2.0
+-----
+
+- `SQL` source improvements:
+  - Deprecate use_cursor in favor of block query construct. The source could previously be configured with:
+
+    ```ruby
+    source Kiba::Pro::Sources::SQL,
+      query: "SELECT * FROM items",
+      use_cursor: true
+    ```
+
+    The `use_cursor` keyword is now deprecated. You can use the more powerful block query construct:
+
+    ```ruby
+    source Kiba::Pro::Sources::SQL,
+      query: -> (db) { db["SELECT * FROM items"].use_cursor },
+    ```
+
+  - Avoid bogus nested SQL calls when configuring the query via block/proc. A call with:
+  
+    ```ruby
+    source Kiba::Pro::Sources::SQL,
+      query: -> (db) { db["SELECT * FROM items"] },
+    ```
+    
+    would have previously generated a `SELECT * FROM (SELECT * FROM "items")`. This is now fixed.
+
+  - Add specs around streaming support (for both MySQL and Postgres).
+  
+    For Postgres, streaming was [recommended by the author of Sequel](https://groups.google.com/d/msg/sequel-talk/olznPcmEf8M/hd5Ris0pYNwJ) over `use_cursor: true` (but do compare on your actual cases!). To enable streaming for Postgres:
+    - Add `sequel_pg` to your `Gemfile`
+    - Enable the extension in your `db` instance & add `.stream` to your dataset e.g.:
+    
+    ```ruby
+    Sequel.connect(ENV.fetch('DATABASE_URL')) do |db|
+      db.extension(:pg_streaming)
+      Kiba.run(Kiba.parse do
+        source Kiba::Pro::Sources::SQL,
+          db: db,
+          query: -> (db) { db[:items].stream }
+        # SNIP
+      end)
+    ```
+    
+    For MySQL, just add `.stream` to your dataset like above (no extension required).
+
+1.1.0
+-----
+
+- Improvement: `SQLBulkInsert` now supports Postgres `INSERT ON CONFLICT` for batch operations (bulk upsert, conditional upserts, ignore if exist etc) via new `dataset` keyword. See [documentation](https://github.com/thbar/kiba/wiki/SQL-Bulk-Insert-Destination).
+
+1.0.0
+-----
 
 NOTE: documentation & requirements/compatibility are available on the [wiki](https://github.com/thbar/kiba/wiki).
 

data/README.md

@@ -1,51 +1,31 @@
-**If you need help**, please [ask your question with tag kiba-etl on StackOverflow](http://stackoverflow.com/questions/ask?tags=kiba-etl) so that other can benefit from your contribution! I monitor this specific tag and will reply to you.
-
-Writing reliable, concise, well-tested & maintainable data-processing code is tricky.
-
-Kiba lets you define and run such high-quality ETL ([Extract-Transform-Load](http://en.wikipedia.org/wiki/Extract,_transform,_load)) jobs using Ruby.
-
-Learn more on the [Wiki](https://github.com/thbar/kiba/wiki), on my [blog](http://thibautbarrere.com) and on [StackOverflow](http://stackoverflow.com/questions/tagged/kiba-etl).
+# Kiba ETL
 
 [![Gem Version](https://badge.fury.io/rb/kiba.svg)](http://badge.fury.io/rb/kiba)
-[![Build Status](https://travis-ci.org/thbar/kiba.svg?branch=master)](https://travis-ci.org/thbar/kiba) [![Build status](https://ci.appveyor.com/api/projects/status/v05jcyhpp1mueq9i?svg=true)](https://ci.appveyor.com/project/thbar/kiba) [![Code Climate](https://codeclimate.com/github/thbar/kiba/badges/gpa.svg)](https://codeclimate.com/github/thbar/kiba) [![Dependency Status](https://gemnasium.com/thbar/kiba.svg)](https://gemnasium.com/thbar/kiba)
+[![Build Status](https://travis-ci.org/thbar/kiba.svg?branch=master)](https://travis-ci.org/thbar/kiba) [![Build status](https://ci.appveyor.com/api/projects/status/v05jcyhpp1mueq9i?svg=true)](https://ci.appveyor.com/project/thbar/kiba) [![Code Climate](https://codeclimate.com/github/thbar/kiba/badges/gpa.svg)](https://codeclimate.com/github/thbar/kiba)
 
-## Note on upcoming Kiba 2.0.0
+Writing reliable, concise, well-tested & maintainable data-processing code is tricky.
 
-Kiba 2.0.0 (available on `master`) includes an improved engine called the `StreamingRunner`, which allows transforms to generate more than one output row for each input row. See [#44](https://github.com/thbar/kiba/pull/44) for documentation on benefits & how to activate.
+Kiba lets you define and run such high-quality ETL ([Extract-Transform-Load](http://en.wikipedia.org/wiki/Extract,_transform,_load)) jobs using Ruby.
 
 ## Getting Started
 
-* [How do you define ETL jobs with Kiba?](https://github.com/thbar/kiba/wiki/How-do-you-define-ETL-jobs-with-Kiba%3F)
-* [How do you run your ETL jobs?](https://github.com/thbar/kiba/wiki/How-do-you-run-your-ETL-jobs%3F)
-* [Implementing ETL sources](https://github.com/thbar/kiba/wiki/Implementing-ETL-sources).
-* [Implementing ETL transforms](https://github.com/thbar/kiba/wiki/Implementing-ETL-transforms).
-* [Implementing ETL destinations](https://github.com/thbar/kiba/wiki/Implementing-ETL-destinations).
-* [Implementing pre and post-processors](https://github.com/thbar/kiba/wiki/Implementing-pre-and-post-processors).
-
-## Useful links
+Head over to the [Wiki](https://github.com/thbar/kiba/wiki) for up-to-date documentation.
 
-* [Live Coding Session - Processing data with Kiba ETL](http://thibautbarrere.com/2015/11/09/video-processing-data-with-kiba-etl/)
-* [Rubyists - are you doing ETL unknowningly?](http://thibautbarrere.com/2015/03/25/rubyists-are-you-doing-etl-unknowingly/)
-* [How to write solid data processing code](http://thibautbarrere.com/2015/04/05/how-to-write-solid-data-processing-code/)
-* [How to reformat CSV files with Kiba](http://thibautbarrere.com/2015/06/04/how-to-reformat-csv-files-with-kiba/) (in-depth, hands-on tutorial)
-* [How to explode multivalued attributes with Kiba ETL?](http://thibautbarrere.com/2015/06/25/how-to-explode-multivalued-attributes-with-kiba/)
-* [Common techniques to compute aggregates with Kiba](https://stackoverflow.com/questions/31145715/how-to-do-a-aggregation-transformation-in-a-kiba-etl-script-kiba-gem)
-* [How to run Kiba in a Rails environment?](http://thibautbarrere.com/2015/09/26/how-to-run-kiba-in-a-rails-environment/)
-* [How to pass parameters to the Kiba command line?](http://stackoverflow.com/questions/32959692/how-to-pass-parameters-into-your-etl-job)
+**If you need help**, please [ask your question with tag kiba-etl on StackOverflow](http://stackoverflow.com/questions/ask?tags=kiba-etl) so that other can benefit from your contribution! I monitor this specific tag and will reply to you.
 
-## Supported Ruby versions
+[Kiba Pro](https://www.kiba-etl.org/kiba-pro) customers get priority private email support for any unforeseen issues and simple matters such as installation troubles. Our consulting services will also be prioritized to Kiba Pro subscribers. If you need any coaching on ETL & data pipeline implementation, please [reach out via email](mailto:info@logeek.fr) so we can discuss how to help you out.
 
-Kiba currently supports Ruby 2.0+ and JRuby (with its default 1.9 syntax). See [test matrix](https://travis-ci.org/thbar/kiba).
+You can also check out the [author blog](https://thibautbarrere.com) and [StackOverflow answers](http://stackoverflow.com/questions/tagged/kiba-etl).
 
-## Kiba Common
+## Supported Ruby versions
 
-I'm starting to add commonly used reusable helpers in a separate gem called [kiba-common](https://github.com/thbar/kiba-common), check it out (work-in-progress).
+Kiba currently supports Ruby 2.4+, JRuby 9.2+ and TruffleRuby. See [test matrix](https://travis-ci.org/thbar/kiba).
 
 ## ETL consulting & commercial version
 
-**Consulting services**: if your organization needs help to implement a data pipeline or to build a data-intensive application, I provide consulting services. [More information](http://thibautbarrere.com/hire-me/).
+**Consulting services**: if your organization needs guidance on Kiba / ETL implementations, we provide consulting services. Contact at [https://www.logeek.fr](https://www.logeek.fr).
 
-**Kiba Pro**: for more features & goodies, check out Kiba Pro ([Changelog & contact info](Pro-Changes.md)).
+**Kiba Pro**: for vendor-backed ETL extensions, check out [Kiba Pro](https://www.kiba-etl.org/kiba-pro).
 
 ## License
 

data/Rakefile

@@ -4,4 +4,9 @@ Rake::TestTask.new(:test) do |t|
   t.pattern = 'test/test_*.rb'
 end
 
-task default: :test
+# A simple check to verify TruffleRuby installation trick is really in effect
+task :show_ruby_version do
+  puts "Running with #{RUBY_DESCRIPTION}"
+end
+
+task default: [:show_ruby_version, :test]

data/appveyor.yml

@@ -5,10 +5,13 @@ cache:
 
 environment:
   matrix:
+    # TODO: add RUBY_VERSION=30 when available (https://www.appveyor.com/updates/)
+    - RUBY_VERSION: 27
+    - RUBY_VERSION: 26
+    - RUBY_VERSION: 25
     - RUBY_VERSION: 24
-    - RUBY_VERSION: 23
-    - RUBY_VERSION: 22
-    - RUBY_VERSION: 21
+    # NOTE: jruby doesn't seem to be supported on default images
+    # see https://www.appveyor.com/docs/build-environment/#ruby
 
 install:
   - set PATH=C:\Ruby%RUBY_VERSION%\bin;%PATH%

data/bin/kiba

@@ -1,5 +1,15 @@
 #!/usr/bin/env ruby
 
-require_relative '../lib/kiba/cli'
+STDERR.puts <<DOC
 
-Kiba::Cli.run(ARGV)
+##########################################################################
+
+The 'kiba' CLI is deprecated and has been removed in Kiba ETL v3.
+
+See release notes / changelog for help.
+
+##########################################################################
+
+DOC
+
+exit(1)

data/kiba.gemspec

@@ -13,6 +13,10 @@ Gem::Specification.new do |gem|
   gem.require_paths = ['lib']
   gem.version       = Kiba::VERSION
   gem.executables   = ['kiba']
+  gem.metadata      = {
+    'source_code_uri'   => 'https://github.com/thbar/kiba',
+    'documentation_uri' => 'https://github.com/thbar/kiba/wiki',
+  }
 
   gem.add_development_dependency 'rake'
   gem.add_development_dependency 'minitest', '~> 5.9'

data/lib/kiba.rb

@@ -11,9 +11,15 @@ require 'kiba/dsl_extensions/config'
 Kiba.extend(Kiba::Parser)
 
 module Kiba
-  def self.run(job)
+  def self.run(job = nil, &block)
+    unless (job.nil? ^ block.nil?)
+      fail ArgumentError.new("Kiba.run takes either one argument (the job) or a block (defining the job)")
+    end
+
+    job ||= Kiba.parse { instance_exec(&block) }
+
     # NOTE: use Hash#dig when Ruby 2.2 reaches EOL
-    runner = job.config.fetch(:kiba, {}).fetch(:runner, Kiba::Runner)
+    runner = job.config.fetch(:kiba, {}).fetch(:runner, Kiba::StreamingRunner)
     runner.run(job)
   end
 end

data/lib/kiba/context.rb

@@ -23,5 +23,9 @@ module Kiba
     def post_process(&block)
       @control.post_processes << { block: block }
     end
+
+    [:source, :transform, :destination].each do |m|
+      ruby2_keywords(m) if respond_to?(:ruby2_keywords, true)
+    end
   end
 end

data/lib/kiba/parser.rb

@@ -1,26 +1,10 @@
-# NOTE: using the "Kiba::Parser" declaration, as I discovered,
-# provides increased isolation to the declared ETL script, compared
-# to 2 nested modules.
-# Before that, a user creating entities named Control, Context
-# or DSLExtensions would see a conflict with Kiba own classes,
-# as by default instance_eval will resolve references by adding
-# the module containing the parser class (initially "Kiba").
-# Now, the classes appear to be further hidden from the user,
-# as Kiba::Parser is its own module.
-# This allows the user to create a Parser, Context, Control class
-# without it being interpreted as reopening Kiba::Parser, Kiba::Context,
-# etc.
-# See test in test_cli.rb (test_namespace_conflict)
-module Kiba::Parser
-  def parse(source_as_string = nil, source_file = nil, &source_as_block)
-    control = Kiba::Control.new
-    context = Kiba::Context.new(control)
-    if source_as_string
-      # this somewhat weird construct allows to remove a nil source_file
-      context.instance_eval(*[source_as_string, source_file].compact)
-    else
+module Kiba
+  module Parser
+    def parse(&source_as_block)
+      control = Kiba::Control.new
+      context = Kiba::Context.new(control)
       context.instance_eval(&source_as_block)
+      control
     end
-    control
   end
 end

data/lib/kiba/runner.rb

@@ -8,9 +8,6 @@ module Kiba
     end
 
     def run(control)
-      # TODO: add a dry-run (not instantiating mode) to_instances call
-      # that will validate the job definition from a syntax pov before
-      # going any further. This could be shared with the parser.
       run_pre_processes(control)
       process_rows(
         to_instances(control.sources),
@@ -18,8 +15,6 @@ module Kiba
         destinations = to_instances(control.destinations)
       )
       close_destinations(destinations)
-      # TODO: when I add post processes as class, I'll have to add a test to
-      # make sure instantiation occurs after the main processing is done (#16)
       run_post_processes(control)
     end
 
@@ -63,15 +58,16 @@ module Kiba
     end
 
     def to_instance(klass, args, block, allow_block, allow_class)
-      if klass
+      if klass && block
+        fail 'Class and block form cannot be used together at the moment'
+      elsif klass
         fail 'Class form is not allowed here' unless allow_class
         klass.new(*args)
       elsif block
         fail 'Block form is not allowed here' unless allow_block
         AliasingProc.new(&block)
       else
-        # TODO: support block passing to a class form definition?
-        fail 'Class and block form cannot be used together at the moment'
+        fail 'Nil parameters not allowed here'
       end
     end
   end

data/lib/kiba/streaming_runner.rb

@@ -11,6 +11,11 @@ module Kiba
           end
           y << returned_row if returned_row
         end
+        if t.respond_to?(:close)
+          t.close do |close_row|
+            y << close_row
+          end
+        end
       end
     end
     

data/lib/kiba/version.rb

@@ -1,3 +1,3 @@
 module Kiba
-  VERSION = '2.0.0.rc1'
+  VERSION = '3.6.0'
 end

data/test/helper.rb

@@ -3,9 +3,11 @@ require 'minitest/pride'
 require 'minitest/focus'
 require 'kiba'
 
-class Kiba::Test < Minitest::Test
-  extend Minitest::Spec::DSL
+if ENV['CI'] == 'true'
+  puts "Running with MiniTest version #{MiniTest::VERSION}"
+end
 
+class Kiba::Test < Minitest::Test
   def remove_files(*files)
     files.each do |file|
       File.delete(file) if File.exist?(file)
@@ -15,4 +17,10 @@ class Kiba::Test < Minitest::Test
   def fixture(file)
     File.join(File.dirname(__FILE__), 'fixtures', file)
   end
+  
+  unless self.method_defined?(:assert_mock)
+    def assert_mock(mock)
+      mock.verify
+    end
+  end
 end

data/test/{common/runner.rb → shared_runner_tests.rb}

@@ -1,11 +1,8 @@
 require 'minitest/mock'
-require_relative '../support/test_enumerable_source'
+require_relative 'support/test_enumerable_source'
+require_relative 'support/test_destination_returning_nil'
 
 module SharedRunnerTests
-  def kiba_run(job)
-    Kiba.run(job)
-  end
-
   def rows
     @rows ||= [
       { identifier: 'first-row' },
@@ -134,4 +131,98 @@ module SharedRunnerTests
     # and the second row should have been reformatted
     assert_equal [{new_identifier: 'second-row'}], @remaining_rows
   end
-end
+  
+  def test_destination_returning_nil_does_not_remove_row_from_pipeline
+    # safeguard to avoid modification on the support code
+    assert_nil TestDestinationReturningNil.new.write("FOOBAR")
+
+    destinations = []
+    control = Kiba.parse do
+      source TestEnumerableSource, [{key: 'value'}]
+      2.times do
+        destination TestDestinationReturningNil, on_init: lambda { |d| destinations << d }
+      end
+    end
+    kiba_run(control)
+    2.times do |i|
+      assert_equal [{key: 'value'}], destinations[i].instance_variable_get(:@written_rows)
+    end
+  end
+
+  def test_nil_transform_error_message
+    control = Kiba.parse do
+      transform
+    end
+    assert_raises(RuntimeError, 'Nil parameters not allowed here') { kiba_run(control) }
+  end
+
+  def test_ruby_3_source_kwargs
+    # NOTE: before Ruby 3 kwargs support, a Ruby warning would 
+    # be captured here with Ruby 2.7 & ensure we fail,
+    # and an error would be raised with Ruby 2.8.0-dev
+    # NOTE: only the first warning will be captured, though, but
+    # having 3 different tests is still better
+    storage = nil
+    assert_silent do
+      Kiba.run(Kiba.parse do
+        source TestKeywordArgumentsComponent,
+          mandatory: "first",
+          on_init: -> (values) { storage = values }
+      end)
+    end
+    assert_equal({
+      mandatory: "first",
+      optional: nil
+    }, storage)
+  end
+  
+  def test_ruby_3_transform_kwargs
+    storage = nil
+    assert_silent do
+      Kiba.run(Kiba.parse do
+        transform TestKeywordArgumentsComponent,
+          mandatory: "first",
+          on_init: -> (values) { storage = values }
+      end)
+    end
+    assert_equal({
+      mandatory: "first",
+      optional: nil
+    }, storage)
+  end
+  
+  def test_ruby_3_destination_kwargs
+    storage = nil
+    assert_silent do
+      Kiba.run(Kiba.parse do
+        destination TestKeywordArgumentsComponent,
+          mandatory: "first",
+          on_init: -> (values) { storage = values }
+      end)
+    end
+    assert_equal({
+      mandatory: "first",
+      optional: nil
+    }, storage)
+  end
+      
+  def test_positional_plus_keyword_arguments
+    storage = nil
+    assert_silent do
+      Kiba.run(Kiba.parse do
+        source TestMixedArgumentsComponent,
+          "some positional argument",
+          mandatory: "first",
+          on_init: -> (values) {
+            storage = values
+          }
+      end)
+    end
+    
+    assert_equal({
+      some_value: "some positional argument",
+      mandatory: "first",
+      optional: nil
+    }, storage)
+  end
+end