kiba 3.0.0 → 3.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a5cf22f1013b65b59d9c91b9f629749b47b5474b3f500d5ca54118b6b6a46c4f
-  data.tar.gz: cae5e05fabc7c8d169d47d6aa6a7da8befd6711ad2d2471c2c0f5f1d3e604841
+  metadata.gz: f8569f0d68410f27d42f38dcb8182ef9e8f47a8c88f714dc64382943febad2c8
+  data.tar.gz: 4d7c6f8c5ef9256182cc51f85c614e383586dd600167e12190c110693d6963fb
 SHA512:
-  metadata.gz: 2ed99641ab799ee57f213525f6ca3ab4b9a9d5e0b7680115bef8e139d32dcfb350fb0117065b5be0ded6a886c21988413b2963e2cbea546a7a1d5691ed078f27
-  data.tar.gz: 24434d65dcf9bb81ce0f575860edbdf92598a1e990663347a599eb42155211c02b9c9eb0304c869cf85c8cd15df8c262afcf9cdf11bb7568323ea98b393b9518
+  metadata.gz: 9fb3c4fd7836e2863771a0086e98a7d8bf24f4df99310464dad6cbc84fef3963155cdac8af0d9a281b6470686334b163491ae9d15158fb19cd5269ef77102129
+  data.tar.gz: 25082194faf157dfa2398485e4cd993ba491871007a3a47c3b5f13db57e677f2f6b11621c2ec5e03ccdce5b44bb6a51e3bfee05968c8a72c99088dfbeb46cd7c

data/.github/FUNDING.yml

@@ -0,0 +1 @@
+github: thbar

data/.travis.yml

@@ -3,6 +3,7 @@ matrix:
   include:
     # see https://www.ruby-lang.org/en/downloads/branches/
     - rvm: ruby-head
+    - rvm: 3.0
     - rvm: 2.7
     - rvm: 2.6
     - rvm: 2.5

data/COMM-LICENSE.md

@@ -177,7 +177,7 @@ The Licensee is responsible for making and keeping adequate backup copies of dat
 
 An active Agreement gives access to “Priority Support”. 
 
-Priority Support covers one (1) email request per quarter. LoGeek will use reasonable endeavours to answer within two (2) French working days. Scope is limited to Kiba and Kiba Pro features and APIs, not the application or infrastructure. For support, email thibaut.barrere+kiba@gmail.com. Please email using the same domain as the original license email or explain your connection to the licensed company. 
+Priority Support covers one (1) email request per quarter. LoGeek will use reasonable endeavours to answer within two (2) French working days. Scope is limited to Kiba and Kiba Pro features and APIs, not the application or infrastructure. For support, email support@logeek.fr. Please email using the same domain as the original license email or explain your connection to the licensed company. 
 
 ## ARTICLE 5 - CUSTOM SUPPORT AND MAINTENANCE
 
@@ -245,7 +245,7 @@ LoGeek is responsible for processing personal data regarding managing contractua
 
 Personal data collected by LoGeek is needed for its processing and is intended for LoGeek’s relevant departments and, where appropriate, its subcontractors and co-contractors, for the requirements of executing the Agreement.
 
-Pursuant to the legal provisions regarding protecting personal data, and under the conditions and to the extent provided by the applicable regulation, each Licensee’s employee concerned shall have a right to query, access, rectify, oppose, obtain erasure or restriction of processing   regarding its personal data – rights which may be exercised by mail sent to the attention of thibaut.barrere+kiba@gmail.com, accompanied, if deemed appropriate by Logeek, by a copy of the relevant person’s identification papers.
+Pursuant to the legal provisions regarding protecting personal data, and under the conditions and to the extent provided by the applicable regulation, each Licensee’s employee concerned shall have a right to query, access, rectify, oppose, obtain erasure or restriction of processing   regarding its personal data – rights which may be exercised by mail sent to the attention of support@logeek.fr, accompanied, if deemed appropriate by Logeek, by a copy of the relevant person’s identification papers.
 
 ## ARTICLE 10 - AUDIT
 

data/Changes.md

@@ -1,3 +1,16 @@
+HEAD
+----
+
+3.6.0
+-----
+
+- New: `Kiba.run(job)` can now (instead of a job parameter) take a block to define the job. See [#94](https://github.com/thbar/kiba/pull/94) for more details.
+
+3.5.0
+-----
+
+- Support for Ruby 2.7+ [#93](https://github.com/thbar/kiba/pull/93). Special thanks to @eregon and @mame for their input.
+
 3.0.0
 -----
 

data/Pro-Changes.md

@@ -1,7 +1,18 @@
 Kiba Pro Changelog
 ==================
 
-Kiba Pro is the commercial extension for Kiba. Documentation is available on the [Wiki](https://github.com/thbar/kiba/wiki).
+Kiba Pro provides vendor-supported ETL extensions for Kiba. Your subscription funds the Open-Source development, thanks for considering it!
+
+Learn more on the [Kiba website](https://www.kiba-etl.org/kiba-pro).
+
+Documentation is available on the [Wiki](https://github.com/thbar/kiba/wiki#kiba-pro).
+
+2.0.0
+-----
+
+- New: `SQLBulkLookup` transform allows to efficiently lookup values in SQL tables. This is particularly useful in datawarehouse scenarios (to replace unique business keys by surrogate keys), or when writing migrations of SQL databases. Instead of looking-up each row individually, it avoids a "N+1" like effect, by working on large batches of rows.
+- New: `ParallelTransform` provides an easy way to process a group of ETL rows at the same time using a pool of threads. It can be used to accelerate ETL transforms doing IO operations such as HTTP queries, by going multithreaded.
+- New: `FileLock` adds an easy way to avoid overlapping runs in ETL Jobs using a local file lock.
 
 1.5.0
 -----

data/README.md

@@ -1,67 +1,31 @@
-**If you need help**, please [ask your question with tag kiba-etl on StackOverflow](http://stackoverflow.com/questions/ask?tags=kiba-etl) so that other can benefit from your contribution! I monitor this specific tag and will reply to you.
-
-Writing reliable, concise, well-tested & maintainable data-processing code is tricky.
-
-Kiba lets you define and run such high-quality ETL ([Extract-Transform-Load](http://en.wikipedia.org/wiki/Extract,_transform,_load)) jobs using Ruby.
-
-Learn more on the [Wiki](https://github.com/thbar/kiba/wiki), on my [blog](http://thibautbarrere.com) and on [StackOverflow](http://stackoverflow.com/questions/tagged/kiba-etl).
-
-A new [kiba-blueprints](https://github.com/thbar/kiba-blueprints) repo has been also created to showcase the use of Kiba OSS and Kiba Pro. More examples are planned!
+# Kiba ETL
 
 [![Gem Version](https://badge.fury.io/rb/kiba.svg)](http://badge.fury.io/rb/kiba)
 [![Build Status](https://travis-ci.org/thbar/kiba.svg?branch=master)](https://travis-ci.org/thbar/kiba) [![Build status](https://ci.appveyor.com/api/projects/status/v05jcyhpp1mueq9i?svg=true)](https://ci.appveyor.com/project/thbar/kiba) [![Code Climate](https://codeclimate.com/github/thbar/kiba/badges/gpa.svg)](https://codeclimate.com/github/thbar/kiba)
 
-## Kiba 2.0.0
-
-Kiba 2.0.0 has been recently released with one major improvement for ETL components developers.
-
-Please check out the [release notes](https://github.com/thbar/kiba/releases/tag/v2.0.0).
+Writing reliable, concise, well-tested & maintainable data-processing code is tricky.
 
-[Subscribe to my newsletter](https://tinyletter.com/kiba-etl) to get the upcoming detailed article on Kiba 2 benefits.
+Kiba lets you define and run such high-quality ETL ([Extract-Transform-Load](http://en.wikipedia.org/wiki/Extract,_transform,_load)) jobs using Ruby.
 
-## Installation
+## Getting Started
 
-Add the gem to your `Gemfile` and run the bundle command:
+Head over to the [Wiki](https://github.com/thbar/kiba/wiki) for up-to-date documentation.
 
-```ruby
-gem 'kiba'
-```
+**If you need help**, please [ask your question with tag kiba-etl on StackOverflow](http://stackoverflow.com/questions/ask?tags=kiba-etl) so that other can benefit from your contribution! I monitor this specific tag and will reply to you.
 
-## Getting Started
+[Kiba Pro](https://www.kiba-etl.org/kiba-pro) customers get priority private email support for any unforeseen issues and simple matters such as installation troubles. Our consulting services will also be prioritized to Kiba Pro subscribers. If you need any coaching on ETL & data pipeline implementation, please [reach out via email](mailto:info@logeek.fr) so we can discuss how to help you out.
 
-* [How do you define ETL jobs with Kiba?](https://github.com/thbar/kiba/wiki/How-do-you-define-ETL-jobs-with-Kiba%3F)
-* [Running Kiba jobs from the command line](https://github.com/thbar/kiba/wiki/Running-Kiba-jobs-from-the-command-line)
-* [Considerations for running Kiba jobs programmatically (from Sidekiq, Faktory, Rake, ...)](https://github.com/thbar/kiba/wiki/Considerations-for-running-Kiba-jobs-programmatically-(from-Sidekiq,-Faktory,-Rake,-...))
-* [Implementing ETL sources](https://github.com/thbar/kiba/wiki/Implementing-ETL-sources).
-* [Implementing ETL transforms](https://github.com/thbar/kiba/wiki/Implementing-ETL-transforms).
-* [Implementing ETL destinations](https://github.com/thbar/kiba/wiki/Implementing-ETL-destinations).
-* [Implementing pre and post-processors](https://github.com/thbar/kiba/wiki/Implementing-pre-and-post-processors).
-
-## Useful links
-
-* [Ruby Kaigi 2018 conference - Kiba 2 - Past, present & future of data processing with Ruby](https://www.youtube.com/watch?v=fxVtbog7pIQ) ([slides](https://speakerdeck.com/thbar/kiba-etl-v2-rubykaigi-2018))
-* [Live Coding Session - Processing data with Kiba ETL](http://thibautbarrere.com/2015/11/09/video-processing-data-with-kiba-etl/)
-* [Rubyists - are you doing ETL unknowningly?](http://thibautbarrere.com/2015/03/25/rubyists-are-you-doing-etl-unknowingly/)
-* [How to write solid data processing code](http://thibautbarrere.com/2015/04/05/how-to-write-solid-data-processing-code/)
-* [How to reformat CSV files with Kiba](http://thibautbarrere.com/2015/06/04/how-to-reformat-csv-files-with-kiba/) (in-depth, hands-on tutorial)
-* [How to explode multivalued attributes with Kiba ETL?](http://thibautbarrere.com/2015/06/25/how-to-explode-multivalued-attributes-with-kiba/)
-* [Common techniques to compute aggregates with Kiba](https://stackoverflow.com/questions/31145715/how-to-do-a-aggregation-transformation-in-a-kiba-etl-script-kiba-gem)
-* [How to run Kiba in a Rails environment?](http://thibautbarrere.com/2015/09/26/how-to-run-kiba-in-a-rails-environment/)
-* [How to pass parameters to the Kiba command line?](http://stackoverflow.com/questions/32959692/how-to-pass-parameters-into-your-etl-job)
+You can also check out the [author blog](https://thibautbarrere.com) and [StackOverflow answers](http://stackoverflow.com/questions/tagged/kiba-etl).
 
 ## Supported Ruby versions
 
-Kiba currently supports Ruby 2.3+, JRuby 9.1+ and TruffleRuby. See [test matrix](https://travis-ci.org/thbar/kiba).
-
-## Kiba Common
-
-I'm starting to add commonly used reusable helpers in a separate gem called [kiba-common](https://github.com/thbar/kiba-common), check it out (work-in-progress).
+Kiba currently supports Ruby 2.4+, JRuby 9.2+ and TruffleRuby. See [test matrix](https://travis-ci.org/thbar/kiba).
 
 ## ETL consulting & commercial version
 
-**Consulting services**: if your organization needs help to implement a data pipeline or to build a data-intensive application, I provide consulting services. [More information](http://thibautbarrere.com/hire-me/).
+**Consulting services**: if your organization needs guidance on Kiba / ETL implementations, we provide consulting services. Contact at [https://www.logeek.fr](https://www.logeek.fr).
 
-**Kiba Pro**: for more features & goodies, check out [Kiba Pro](https://github.com/thbar/kiba/wiki#kiba-pro).
+**Kiba Pro**: for vendor-backed ETL extensions, check out [Kiba Pro](https://www.kiba-etl.org/kiba-pro).
 
 ## License
 

data/appveyor.yml

@@ -5,10 +5,11 @@ cache:
 
 environment:
   matrix:
+    # TODO: add RUBY_VERSION=30 when available (https://www.appveyor.com/updates/)
+    - RUBY_VERSION: 27
     - RUBY_VERSION: 26
     - RUBY_VERSION: 25
     - RUBY_VERSION: 24
-    - RUBY_VERSION: 23
     # NOTE: jruby doesn't seem to be supported on default images
     # see https://www.appveyor.com/docs/build-environment/#ruby
 

data/lib/kiba.rb

@@ -11,7 +11,13 @@ require 'kiba/dsl_extensions/config'
 Kiba.extend(Kiba::Parser)
 
 module Kiba
-  def self.run(job)
+  def self.run(job = nil, &block)
+    unless (job.nil? ^ block.nil?)
+      fail ArgumentError.new("Kiba.run takes either one argument (the job) or a block (defining the job)")
+    end
+
+    job ||= Kiba.parse { instance_exec(&block) }
+
     # NOTE: use Hash#dig when Ruby 2.2 reaches EOL
     runner = job.config.fetch(:kiba, {}).fetch(:runner, Kiba::StreamingRunner)
     runner.run(job)

data/lib/kiba/context.rb

@@ -23,5 +23,9 @@ module Kiba
     def post_process(&block)
       @control.post_processes << { block: block }
     end
+
+    [:source, :transform, :destination].each do |m|
+      ruby2_keywords(m) if respond_to?(:ruby2_keywords, true)
+    end
   end
 end

data/lib/kiba/runner.rb

@@ -8,9 +8,6 @@ module Kiba
     end
 
     def run(control)
-      # TODO: add a dry-run (not instantiating mode) to_instances call
-      # that will validate the job definition from a syntax pov before
-      # going any further. This could be shared with the parser.
       run_pre_processes(control)
       process_rows(
         to_instances(control.sources),
@@ -18,8 +15,6 @@ module Kiba
         destinations = to_instances(control.destinations)
       )
       close_destinations(destinations)
-      # TODO: when I add post processes as class, I'll have to add a test to
-      # make sure instantiation occurs after the main processing is done (#16)
       run_post_processes(control)
     end
 

data/lib/kiba/version.rb

@@ -1,3 +1,3 @@
 module Kiba
-  VERSION = '3.0.0'
+  VERSION = '3.6.0'
 end

data/test/shared_runner_tests.rb

@@ -155,4 +155,74 @@ module SharedRunnerTests
     end
     assert_raises(RuntimeError, 'Nil parameters not allowed here') { kiba_run(control) }
   end
+
+  def test_ruby_3_source_kwargs
+    # NOTE: before Ruby 3 kwargs support, a Ruby warning would 
+    # be captured here with Ruby 2.7 & ensure we fail,
+    # and an error would be raised with Ruby 2.8.0-dev
+    # NOTE: only the first warning will be captured, though, but
+    # having 3 different tests is still better
+    storage = nil
+    assert_silent do
+      Kiba.run(Kiba.parse do
+        source TestKeywordArgumentsComponent,
+          mandatory: "first",
+          on_init: -> (values) { storage = values }
+      end)
+    end
+    assert_equal({
+      mandatory: "first",
+      optional: nil
+    }, storage)
+  end
+  
+  def test_ruby_3_transform_kwargs
+    storage = nil
+    assert_silent do
+      Kiba.run(Kiba.parse do
+        transform TestKeywordArgumentsComponent,
+          mandatory: "first",
+          on_init: -> (values) { storage = values }
+      end)
+    end
+    assert_equal({
+      mandatory: "first",
+      optional: nil
+    }, storage)
+  end
+  
+  def test_ruby_3_destination_kwargs
+    storage = nil
+    assert_silent do
+      Kiba.run(Kiba.parse do
+        destination TestKeywordArgumentsComponent,
+          mandatory: "first",
+          on_init: -> (values) { storage = values }
+      end)
+    end
+    assert_equal({
+      mandatory: "first",
+      optional: nil
+    }, storage)
+  end
+      
+  def test_positional_plus_keyword_arguments
+    storage = nil
+    assert_silent do
+      Kiba.run(Kiba.parse do
+        source TestMixedArgumentsComponent,
+          "some positional argument",
+          mandatory: "first",
+          on_init: -> (values) {
+            storage = values
+          }
+      end)
+    end
+    
+    assert_equal({
+      some_value: "some positional argument",
+      mandatory: "first",
+      optional: nil
+    }, storage)
+  end
 end

data/test/support/test_keyword_arguments_component.rb

@@ -0,0 +1,14 @@
+# a mock component to test Ruby 3 keyword argument support
+class TestKeywordArgumentsComponent
+  def initialize(mandatory:, optional: nil, on_init: nil)
+    values = {
+      mandatory: mandatory,
+      optional: optional
+    }
+    on_init&.call(values)
+  end
+  
+  def each
+    # no-op
+  end
+end

data/test/support/test_mixed_arguments_component.rb

@@ -0,0 +1,14 @@
+# a mock component to test Ruby 3 keyword argument support
+class TestMixedArgumentsComponent
+  def initialize(some_value, mandatory:, optional: nil, on_init:)
+    @values = {}
+    @values[:some_value] = some_value
+    @values[:mandatory] = mandatory
+    @values[:optional] = optional
+    on_init&.call(@values)
+  end
+  
+  def each
+    # no-op
+  end
+end

data/test/test_run.rb

@@ -1,5 +1,7 @@
 require_relative 'helper'
 require 'minitest/mock'
+require_relative 'support/test_enumerable_source'
+require_relative 'support/test_array_destination'
 
 class TestRun < Kiba::Test
   def test_ensure_kiba_defaults_to_streaming_runner
@@ -9,4 +11,28 @@ class TestRun < Kiba::Test
       assert_equal "Streaming runner called", Kiba.run(job)
     end
   end
+
+  def test_run_allows_block_arg
+    rows = []
+    Kiba.run do
+      source TestEnumerableSource, (1..10)
+      destination TestArrayDestination, rows
+    end
+    assert_equal (1..10).to_a, rows
+  end
+
+  def test_forbids_no_arg
+    assert_raises ArgumentError do
+      Kiba.run
+    end
+  end
+
+  def test_forbids_multiple_args
+    assert_raises ArgumentError do
+      job = Kiba.parse { }
+      Kiba.run(job) do
+        #
+      end
+    end
+  end
 end

data/test/test_streaming_runner.rb

@@ -6,6 +6,8 @@ require_relative 'support/test_duplicate_row_transform'
 require_relative 'support/test_close_yielding_transform'
 require_relative 'support/test_non_closing_transform'
 require_relative 'shared_runner_tests'
+require_relative 'support/test_keyword_arguments_component'
+require_relative 'support/test_mixed_arguments_component'
 
 class TestStreamingRunner < Kiba::Test
   def kiba_run(job)

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: kiba
 version: !ruby/object:Gem::Version
-  version: 3.0.0
+  version: 3.6.0
 platform: ruby
 authors:
 - Thibaut Barrère
-autorequire: 
+autorequire:
 bindir: bin
 cert_chain: []
-date: 2020-02-10 00:00:00.000000000 Z
+date: 2021-02-07 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rake
@@ -74,6 +74,7 @@ executables:
 extensions: []
 extra_rdoc_files: []
 files:
+- ".github/FUNDING.yml"
 - ".gitignore"
 - ".travis.yml"
 - COMM-LICENSE.md
@@ -106,6 +107,8 @@ files:
 - test/support/test_destination_returning_nil.rb
 - test/support/test_duplicate_row_transform.rb
 - test/support/test_enumerable_source.rb
+- test/support/test_keyword_arguments_component.rb
+- test/support/test_mixed_arguments_component.rb
 - test/support/test_non_closing_transform.rb
 - test/support/test_rename_field_transform.rb
 - test/support/test_source_that_reads_at_instantiation_time.rb
@@ -122,7 +125,7 @@ licenses:
 metadata:
   source_code_uri: https://github.com/thbar/kiba
   documentation_uri: https://github.com/thbar/kiba/wiki
-post_install_message: 
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -137,8 +140,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.0.3
-signing_key: 
+rubygems_version: 3.2.3
+signing_key:
 specification_version: 4
 summary: Lightweight ETL for Ruby
 test_files:
@@ -153,6 +156,8 @@ test_files:
 - test/support/test_destination_returning_nil.rb
 - test/support/test_duplicate_row_transform.rb
 - test/support/test_enumerable_source.rb
+- test/support/test_keyword_arguments_component.rb
+- test/support/test_mixed_arguments_component.rb
 - test/support/test_non_closing_transform.rb
 - test/support/test_rename_field_transform.rb
 - test/support/test_source_that_reads_at_instantiation_time.rb