kiba 2.0.0 → 4.0.0

data/Gemfile

@@ -1,3 +1,3 @@
-source 'https://rubygems.org'
+source "https://rubygems.org"
 
 gemspec

data/ISSUE_TEMPLATE.md

@@ -0,0 +1,7 @@
+**If you need help**, please:
+* [Check existing answers on StackOverflow](https://stackoverflow.com/questions/tagged/kiba-etl).
+* [Ask your question with tag kiba-etl on StackOverflow](http://stackoverflow.com/questions/ask?tags=kiba-etl) so that other can benefit from your contribution!
+
+I monitor this specific tag and will reply to you.
+
+Please only open an issue in case you found a bug. Thanks!

data/LICENSE

@@ -2,4 +2,6 @@ Copyright (c) LoGeek SARL
 
 Kiba Common is an Open Source project licensed under the terms of
 the LGPLv3 license.  Please see <http://www.gnu.org/licenses/lgpl-3.0.html>
-for license text.
+for license text.
+
+Kiba Pro has a commercial-friendly license allowing private forks and modifications of Kiba. You can find the commercial license terms in COMM-LICENSE.md.

data/Pro-Changes.md

@@ -1,13 +1,90 @@
 Kiba Pro Changelog
 ==================
 
-Kiba Pro is the commercial extension for Kiba. Documentation is available on the [Wiki](https://github.com/thbar/kiba/wiki).
+Kiba Pro provides vendor-supported ETL extensions for Kiba. Your subscription funds the Open-Source development, thanks for considering it!
 
-HEAD
--------
+Learn more on the [Kiba website](https://www.kiba-etl.org/kiba-pro).
 
-1.0.0.rc1
----------
+Documentation is available on the [Wiki](https://github.com/thbar/kiba/wiki#kiba-pro).
+
+2.0.0
+-----
+
+- New: `SQLBulkLookup` transform allows to efficiently lookup values in SQL tables. This is particularly useful in datawarehouse scenarios (to replace unique business keys by surrogate keys), or when writing migrations of SQL databases. Instead of looking-up each row individually, it avoids a "N+1" like effect, by working on large batches of rows.
+- New: `ParallelTransform` provides an easy way to process a group of ETL rows at the same time using a pool of threads. It can be used to accelerate ETL transforms doing IO operations such as HTTP queries, by going multithreaded.
+- New: `FileLock` adds an easy way to avoid overlapping runs in ETL Jobs using a local file lock.
+
+1.5.0
+-----
+
+- Compatibility with Kiba v3
+- BREAKING CHANGE: deprecate non-live Sequel connection passing (https://github.com/thbar/kiba/issues/79). Do not use `database: "connection_string"`, instead pass your `Sequel` connection directly. This moves the connection management out of the destination, which is a better pattern & provides better (block-based) resources closing.
+- Official MySQL support:
+  - While the compatibility was already here, it is now tested for in our QA testing suite.
+  - MySQL 5.5-8.0 is supported & tested
+  - MariaDB should be supported (although not tested against in the QA testing suite)
+  - Amazon Aurora MySQL is also supposed to work (although not tested)
+  - `Kiba::Pro::Sources::SQL` supports for non-streaming + streaming use
+  - `Kiba::Pro::Destinations::SQLBulkInsert` supports:
+    - Bulk insert
+    - Bulk insert with ignore
+    - Bulk upsert (including with dynamically computed columns) via `ON DUPLICATE KEY UPDATE`
+  - Note that the `Kiba::Pro::Destinations::SQLUpsert` (row-by-row) is not MySQL compatible at the moment
+
+1.2.0
+-----
+
+- `SQL` source improvements:
+  - Deprecate use_cursor in favor of block query construct. The source could previously be configured with:
+
+    ```ruby
+    source Kiba::Pro::Sources::SQL,
+      query: "SELECT * FROM items",
+      use_cursor: true
+    ```
+
+    The `use_cursor` keyword is now deprecated. You can use the more powerful block query construct:
+
+    ```ruby
+    source Kiba::Pro::Sources::SQL,
+      query: -> (db) { db["SELECT * FROM items"].use_cursor },
+    ```
+
+  - Avoid bogus nested SQL calls when configuring the query via block/proc. A call with:
+  
+    ```ruby
+    source Kiba::Pro::Sources::SQL,
+      query: -> (db) { db["SELECT * FROM items"] },
+    ```
+    
+    would have previously generated a `SELECT * FROM (SELECT * FROM "items")`. This is now fixed.
+
+  - Add specs around streaming support (for both MySQL and Postgres).
+  
+    For Postgres, streaming was [recommended by the author of Sequel](https://groups.google.com/d/msg/sequel-talk/olznPcmEf8M/hd5Ris0pYNwJ) over `use_cursor: true` (but do compare on your actual cases!). To enable streaming for Postgres:
+    - Add `sequel_pg` to your `Gemfile`
+    - Enable the extension in your `db` instance & add `.stream` to your dataset e.g.:
+    
+    ```ruby
+    Sequel.connect(ENV.fetch('DATABASE_URL')) do |db|
+      db.extension(:pg_streaming)
+      Kiba.run(Kiba.parse do
+        source Kiba::Pro::Sources::SQL,
+          db: db,
+          query: -> (db) { db[:items].stream }
+        # SNIP
+      end)
+    ```
+    
+    For MySQL, just add `.stream` to your dataset like above (no extension required).
+
+1.1.0
+-----
+
+- Improvement: `SQLBulkInsert` now supports Postgres `INSERT ON CONFLICT` for batch operations (bulk upsert, conditional upserts, ignore if exist etc) via new `dataset` keyword. See [documentation](https://github.com/thbar/kiba/wiki/SQL-Bulk-Insert-Destination).
+
+1.0.0
+-----
 
 NOTE: documentation & requirements/compatibility are available on the [wiki](https://github.com/thbar/kiba/wiki).
 

data/README.md

@@ -1,84 +1,31 @@
-**If you need help**, please [ask your question with tag kiba-etl on StackOverflow](http://stackoverflow.com/questions/ask?tags=kiba-etl) so that other can benefit from your contribution! I monitor this specific tag and will reply to you.
-
-Writing reliable, concise, well-tested & maintainable data-processing code is tricky.
-
-Kiba lets you define and run such high-quality ETL ([Extract-Transform-Load](http://en.wikipedia.org/wiki/Extract,_transform,_load)) jobs using Ruby.
-
-Learn more on the [Wiki](https://github.com/thbar/kiba/wiki), on my [blog](http://thibautbarrere.com) and on [StackOverflow](http://stackoverflow.com/questions/tagged/kiba-etl).
+# Kiba ETL
 
 [![Gem Version](https://badge.fury.io/rb/kiba.svg)](http://badge.fury.io/rb/kiba)
-[![Build Status](https://travis-ci.org/thbar/kiba.svg?branch=master)](https://travis-ci.org/thbar/kiba) [![Build status](https://ci.appveyor.com/api/projects/status/v05jcyhpp1mueq9i?svg=true)](https://ci.appveyor.com/project/thbar/kiba) [![Code Climate](https://codeclimate.com/github/thbar/kiba/badges/gpa.svg)](https://codeclimate.com/github/thbar/kiba) [![Dependency Status](https://gemnasium.com/thbar/kiba.svg)](https://gemnasium.com/thbar/kiba)
-
-## Kiba 2.0.0.rc1
-
-Kiba 2.0.0.rc1 (available via `gem install kiba --prerelease`) is available for testing.
-
-### New StreamingRunner engine
-
-Kiba 2 introduces a new, opt-in engine called the `StreamingRunner`, which allows to generate an arbitrary number of rows inside transforms. This drastically improves the reusability & composability of Kiba components (see [#44](https://github.com/thbar/kiba/pull/44) for some background).
-
-To use the `StreamingRunner`, use the following code:
+[![Build Status](https://github.com/thbar/kiba/actions/workflows/ci.yml/badge.svg)](https://github.com/thbar/kiba/actions) [![Code Climate](https://codeclimate.com/github/thbar/kiba/badges/gpa.svg)](https://codeclimate.com/github/thbar/kiba)
 
-```ruby
-# activate the new Kiba internal config system
-extend Kiba::DSLExtensions::Config
-# opt-in for the new engine
-config :kiba, runner: Kiba::StreamingRunner
-
-# write transform class able to yield an arbitrary number of rows
-class MyYieldingTransform
-  def process(row)
-    yield {key: 1}
-    yield {key: 2}
-    {key: 3}
-  end
-end
-```
-
-The improved runner is compatible with Ruby 2.0+.
-
-### Compatibility with Kiba 1
-
-Kiba 2 is expected to be compatible with existing Kiba scripts as long as you did not use internal API.
-
-Internal changes include:
+Writing reliable, concise, well-tested & maintainable data-processing code is tricky.
 
-* An opt-in, Elixir's mix-inspired `config` system, currently only used to select the runner you want at job declaration time
-* A stronger isolation in the `Parser`, to reduces the chances that ETL scripts could conflict with Kiba internal classes
+Kiba lets you define and run such high-quality ETL ([Extract-Transform-Load](http://en.wikipedia.org/wiki/Extract,_transform,_load)) jobs using Ruby.
 
 ## Getting Started
 
-* [How do you define ETL jobs with Kiba?](https://github.com/thbar/kiba/wiki/How-do-you-define-ETL-jobs-with-Kiba%3F)
-* [How do you run your ETL jobs?](https://github.com/thbar/kiba/wiki/How-do-you-run-your-ETL-jobs%3F)
-* [Implementing ETL sources](https://github.com/thbar/kiba/wiki/Implementing-ETL-sources).
-* [Implementing ETL transforms](https://github.com/thbar/kiba/wiki/Implementing-ETL-transforms).
-* [Implementing ETL destinations](https://github.com/thbar/kiba/wiki/Implementing-ETL-destinations).
-* [Implementing pre and post-processors](https://github.com/thbar/kiba/wiki/Implementing-pre-and-post-processors).
+Head over to the [Wiki](https://github.com/thbar/kiba/wiki) for up-to-date documentation.
 
-## Useful links
+**If you need help**, please [ask your question with tag kiba-etl on StackOverflow](http://stackoverflow.com/questions/ask?tags=kiba-etl) so that other can benefit from your contribution! I monitor this specific tag and will reply to you.
 
-* [Live Coding Session - Processing data with Kiba ETL](http://thibautbarrere.com/2015/11/09/video-processing-data-with-kiba-etl/)
-* [Rubyists - are you doing ETL unknowningly?](http://thibautbarrere.com/2015/03/25/rubyists-are-you-doing-etl-unknowingly/)
-* [How to write solid data processing code](http://thibautbarrere.com/2015/04/05/how-to-write-solid-data-processing-code/)
-* [How to reformat CSV files with Kiba](http://thibautbarrere.com/2015/06/04/how-to-reformat-csv-files-with-kiba/) (in-depth, hands-on tutorial)
-* [How to explode multivalued attributes with Kiba ETL?](http://thibautbarrere.com/2015/06/25/how-to-explode-multivalued-attributes-with-kiba/)
-* [Common techniques to compute aggregates with Kiba](https://stackoverflow.com/questions/31145715/how-to-do-a-aggregation-transformation-in-a-kiba-etl-script-kiba-gem)
-* [How to run Kiba in a Rails environment?](http://thibautbarrere.com/2015/09/26/how-to-run-kiba-in-a-rails-environment/)
-* [How to pass parameters to the Kiba command line?](http://stackoverflow.com/questions/32959692/how-to-pass-parameters-into-your-etl-job)
+[Kiba Pro](https://www.kiba-etl.org/kiba-pro) customers get priority private email support for any unforeseen issues and simple matters such as installation troubles. Our consulting services will also be prioritized to Kiba Pro subscribers. If you need any coaching on ETL & data pipeline implementation, please [reach out via email](mailto:info@logeek.fr) so we can discuss how to help you out.
 
-## Supported Ruby versions
+You can also check out the [author blog](https://thibautbarrere.com) and [StackOverflow answers](http://stackoverflow.com/questions/tagged/kiba-etl).
 
-Kiba currently supports Ruby 2.0+ and JRuby (with its default 1.9 syntax). See [test matrix](https://travis-ci.org/thbar/kiba).
-
-## Kiba Common
+## Supported Ruby versions
 
-I'm starting to add commonly used reusable helpers in a separate gem called [kiba-common](https://github.com/thbar/kiba-common), check it out (work-in-progress).
+Kiba currently supports Ruby 2.5+, JRuby 9.2+ and TruffleRuby. See [test matrix](https://github.com/thbar/kiba/actions).
 
 ## ETL consulting & commercial version
 
-**Consulting services**: if your organization needs help to implement a data pipeline or to build a data-intensive application, I provide consulting services. [More information](http://thibautbarrere.com/hire-me/).
+**Consulting services**: if your organization needs guidance on Kiba / ETL implementations, we provide consulting services. Contact at [https://www.logeek.fr](https://www.logeek.fr).
 
-**Kiba Pro**: for more features & goodies, check out Kiba Pro ([Changelog & contact info](Pro-Changes.md)).
+**Kiba Pro**: for vendor-backed ETL extensions, check out [Kiba Pro](https://www.kiba-etl.org/kiba-pro).
 
 ## License
 

data/Rakefile

@@ -1,7 +1,12 @@
-require 'rake/testtask'
+require "rake/testtask"
 
 Rake::TestTask.new(:test) do |t|
-  t.pattern = 'test/test_*.rb'
+  t.pattern = "test/test_*.rb"
 end
 
-task default: :test
+# A simple check to verify TruffleRuby installation trick is really in effect
+task :show_ruby_version do
+  puts "Running with #{RUBY_DESCRIPTION}"
+end
+
+task default: [:show_ruby_version, :test]

data/kiba.gemspec

@@ -1,21 +1,24 @@
-# -*- encoding: utf-8 -*-
-require File.expand_path('../lib/kiba/version', __FILE__)
+require File.expand_path("../lib/kiba/version", __FILE__)
 
 Gem::Specification.new do |gem|
-  gem.authors       = ['Thibaut Barrère']
-  gem.email         = ['thibaut.barrere@gmail.com']
-  gem.description   = gem.summary = 'Lightweight ETL for Ruby'
-  gem.homepage      = 'http://thbar.github.io/kiba/'
-  gem.license       = 'LGPL-3.0'
-  gem.files         = `git ls-files | grep -Ev '^(examples)'`.split("\n")
-  gem.test_files    = `git ls-files -- test/*`.split("\n")
-  gem.name          = 'kiba'
-  gem.require_paths = ['lib']
-  gem.version       = Kiba::VERSION
-  gem.executables   = ['kiba']
+  gem.authors = ["Thibaut Barrère"]
+  gem.email = ["thibaut.barrere@gmail.com"]
+  gem.description = gem.summary = "Lightweight ETL for Ruby"
+  gem.homepage = "https://www.kiba-etl.org"
+  gem.license = "LGPL-3.0"
+  gem.files = `git ls-files | grep -Ev '^(examples)'`.split("\n")
+  gem.test_files = `git ls-files -- test/*`.split("\n")
+  gem.name = "kiba"
+  gem.require_paths = ["lib"]
+  gem.version = Kiba::VERSION
+  gem.metadata = {
+    "source_code_uri" => "https://github.com/thbar/kiba",
+    "documentation_uri" => "https://github.com/thbar/kiba/wiki"
+  }
 
-  gem.add_development_dependency 'rake'
-  gem.add_development_dependency 'minitest', '~> 5.9'
-  gem.add_development_dependency 'awesome_print'
-  gem.add_development_dependency 'minitest-focus'
+  gem.add_development_dependency "rake"
+  gem.add_development_dependency "minitest", "~> 5.9"
+  gem.add_development_dependency "awesome_print"
+  gem.add_development_dependency "minitest-focus"
+  gem.add_development_dependency "standard"
 end

data/lib/kiba.rb

@@ -1,19 +1,22 @@
-# encoding: utf-8
-require 'kiba/version'
+require "kiba/version"
 
-require 'kiba/control'
-require 'kiba/context'
-require 'kiba/parser'
-require 'kiba/runner'
-require 'kiba/streaming_runner'
-require 'kiba/dsl_extensions/config'
+require "kiba/control"
+require "kiba/context"
+require "kiba/parser"
+require "kiba/streaming_runner"
+require "kiba/dsl_extensions/config"
 
 Kiba.extend(Kiba::Parser)
 
 module Kiba
-  def self.run(job)
-    # NOTE: use Hash#dig when Ruby 2.2 reaches EOL
-    runner = job.config.fetch(:kiba, {}).fetch(:runner, Kiba::Runner)
+  def self.run(job = nil, &block)
+    unless job.nil? ^ block.nil?
+      fail ArgumentError.new("Kiba.run takes either one argument (the job) or a block (defining the job)")
+    end
+
+    job ||= Kiba.parse { instance_exec(&block) }
+
+    runner = job.config.fetch(:kiba, {}).fetch(:runner, Kiba::StreamingRunner)
     runner.run(job)
   end
 end

data/lib/kiba/context.rb

@@ -5,23 +5,27 @@ module Kiba
     end
 
     def pre_process(&block)
-      @control.pre_processes << { block: block }
+      @control.pre_processes << {block: block}
     end
 
     def source(klass, *initialization_params)
-      @control.sources << { klass: klass, args: initialization_params }
+      @control.sources << {klass: klass, args: initialization_params}
     end
 
     def transform(klass = nil, *initialization_params, &block)
-      @control.transforms << { klass: klass, args: initialization_params, block: block }
+      @control.transforms << {klass: klass, args: initialization_params, block: block}
     end
 
     def destination(klass, *initialization_params)
-      @control.destinations << { klass: klass, args: initialization_params }
+      @control.destinations << {klass: klass, args: initialization_params}
     end
 
     def post_process(&block)
-      @control.post_processes << { block: block }
+      @control.post_processes << {block: block}
+    end
+
+    [:source, :transform, :destination].each do |m|
+      ruby2_keywords(m) if respond_to?(:ruby2_keywords, true)
     end
   end
 end

data/lib/kiba/control.rb

@@ -3,7 +3,7 @@ module Kiba
     def pre_processes
       @pre_processes ||= []
     end
-    
+
     def config
       @config ||= {}
     end

data/lib/kiba/dsl_extensions/config.rb

@@ -6,4 +6,4 @@ module Kiba
       end
     end
   end
-end
+end

data/lib/kiba/parser.rb

@@ -1,26 +1,10 @@
-# NOTE: using the "Kiba::Parser" declaration, as I discovered,
-# provides increased isolation to the declared ETL script, compared
-# to 2 nested modules.
-# Before that, a user creating entities named Control, Context
-# or DSLExtensions would see a conflict with Kiba own classes,
-# as by default instance_eval will resolve references by adding
-# the module containing the parser class (initially "Kiba").
-# Now, the classes appear to be further hidden from the user,
-# as Kiba::Parser is its own module.
-# This allows the user to create a Parser, Context, Control class
-# without it being interpreted as reopening Kiba::Parser, Kiba::Context,
-# etc.
-# See test in test_cli.rb (test_namespace_conflict)
-module Kiba::Parser
-  def parse(source_as_string = nil, source_file = nil, &source_as_block)
-    control = Kiba::Control.new
-    context = Kiba::Context.new(control)
-    if source_as_string
-      # this somewhat weird construct allows to remove a nil source_file
-      context.instance_eval(*[source_as_string, source_file].compact)
-    else
+module Kiba
+  module Parser
+    def parse(&source_as_block)
+      control = Kiba::Control.new
+      context = Kiba::Context.new(control)
       context.instance_eval(&source_as_block)
+      control
     end
-    control
   end
 end

data/lib/kiba/streaming_runner.rb

@@ -1,8 +1,37 @@
 module Kiba
   module StreamingRunner
-    include Runner
     extend self
-    
+
+    # allow to handle a block form just like a regular transform
+    class AliasingProc < Proc
+      alias_method :process, :call
+    end
+
+    def run(control)
+      run_pre_processes(control)
+      process_rows(
+        to_instances(control.sources),
+        to_instances(control.transforms, true),
+        destinations = to_instances(control.destinations)
+      )
+      close_destinations(destinations)
+      run_post_processes(control)
+    end
+
+    def run_pre_processes(control)
+      to_instances(control.pre_processes, true, false).each(&:call)
+    end
+
+    def run_post_processes(control)
+      to_instances(control.post_processes, true, false).each(&:call)
+    end
+
+    def close_destinations(destinations)
+      destinations
+        .find_all { |d| d.respond_to?(:close) }
+        .each(&:close)
+    end
+
     def transform_stream(stream, t)
       Enumerator.new do |y|
         stream.each do |input_row|
@@ -11,9 +40,14 @@ module Kiba
           end
           y << returned_row if returned_row
         end
+        if t.respond_to?(:close)
+          t.close do |close_row|
+            y << close_row
+          end
+        end
       end
     end
-    
+
     def source_stream(sources)
       Enumerator.new do |y|
         sources.each do |source|
@@ -24,10 +58,33 @@ module Kiba
 
     def process_rows(sources, transforms, destinations)
       stream = source_stream(sources)
-      recurser = lambda { |s,t| transform_stream(s, t) }
+      recurser = lambda { |s, t| transform_stream(s, t) }
       transforms.inject(stream, &recurser).each do |r|
         destinations.each { |d| d.write(r) }
       end
     end
+
+    def to_instances(definitions, allow_block = false, allow_class = true)
+      definitions.map do |definition|
+        to_instance(
+          *definition.values_at(:klass, :args, :block),
+          allow_block, allow_class
+        )
+      end
+    end
+
+    def to_instance(klass, args, block, allow_block, allow_class)
+      if klass && block
+        fail "Class and block form cannot be used together at the moment"
+      elsif klass
+        fail "Class form is not allowed here" unless allow_class
+        klass.new(*args)
+      elsif block
+        fail "Block form is not allowed here" unless allow_block
+        AliasingProc.new(&block)
+      else
+        fail "Nil parameters not allowed here"
+      end
+    end
   end
-end
+end