kiba 0.5.0 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 502470fc246c67daaa681ca78fb5337899cca7fa
-  data.tar.gz: a125ff166156c79e5a0b0d67bf9dfb980b7e0dba
+  metadata.gz: faa6cbb049d4b35cdd62c647c1510bc9d296cbb5
+  data.tar.gz: c6d522da427b0b2388771279b146fd48ac117605
 SHA512:
-  metadata.gz: ccefac21a401ca860d34c89fdda2473e5a30b51d61223fc8cced50165786f41f328014144bd31486522db34c4e801190060d250cad20408745c691ca937ea1ea
-  data.tar.gz: 6c0bee993d99fdec14504e6811549af4dce40cd8930be951a142f8793da69956283ed7ccf6acececa4a6a108e9e9f424d2190b6ba7d9e45f55207b3ee240418d
+  metadata.gz: b26ef4488c4aa78c86f99fb001565260dacae1ef529f2c7b8c37533ef179e39c57dfc8c6e35fa1f31acf8e9e4ab6d22b418d90316317a6f4db898c3a93c22108
+  data.tar.gz: f4d7cc78c3ccdb04fc3b98310ac2f4f9163aa60112481b81f22119f4864c6485117087718ec3512559ac668f62d2f8bcc7b54275999303941d9373225851b620

data/Changes.md

@@ -1,3 +1,11 @@
+Unreleased
+----------
+
+0.6.0
+-----
+
+- Add `pre_process` block support
+
 0.5.0
 -----
 

data/Gemfile

@@ -1,3 +1,3 @@
 source 'https://rubygems.org'
 
-gemspec
+gemspec

data/README.md

@@ -1,9 +1,13 @@
 Writing reliable, concise, well-tested & maintainable data-processing code is tricky.
 
-Kiba lets you define and run such high-quality ETL jobs, using Ruby.
+Kiba lets you define and run such high-quality ETL ([Extract-Transform-Load](http://en.wikipedia.org/wiki/Extract,_transform,_load)) jobs, using Ruby (see [supported versions](#supported-ruby-versions)).
 
-**Note: this is EARLY WORK - the API/syntax may change at any time.**
+Learn more on the [Kiba blog](http://thibautbarrere.com):
 
+* [Rubyists - are you doing ETL unknowningly?](http://thibautbarrere.com/2015/03/25/rubyists-are-you-doing-etl-unknowingly/)
+* [How to write solid data processing code](http://thibautbarrere.com/2015/04/05/how-to-write-solid-data-processing-code/)
+
+[![Gem Version](https://badge.fury.io/rb/kiba.svg)](http://badge.fury.io/rb/kiba)
 [![Build Status](https://travis-ci.org/thbar/kiba.svg?branch=master)](https://travis-ci.org/thbar/kiba) [![Code Climate](https://codeclimate.com/github/thbar/kiba/badges/gpa.svg)](https://codeclimate.com/github/thbar/kiba) [![Dependency Status](https://gemnasium.com/thbar/kiba.svg)](https://gemnasium.com/thbar/kiba)
 
 ## How do you define ETL jobs with Kiba?
@@ -20,6 +24,11 @@ end
 # eg: commonly used sources / destinations / transforms, under unit-test
 require_relative 'common'
 
+# declare a pre-processor: a block called before the first row is read
+pre_process do
+  # do something
+end
+
 # declare a source where to take data from (you implement it - see notes below)
 source MyCsvSource, 'input.csv'
 
@@ -50,7 +59,7 @@ post_process do
 end
 ```
 
-The combination of sources, transforms, destinations and post-processors defines the data processing pipeline.
+The combination of pre-processors, sources, transforms, destinations and post-processors defines the data processing pipeline.
 
 Note: you are advised to store your ETL definitions as files with the extension `.etl` (rather than `.rb`). This will make sure you do not end up loading them by mistake from another component (eg: a Rails app).
 
@@ -74,7 +83,7 @@ Kiba.run(job_definition)
 `Kiba.parse` evaluates your ETL Ruby code to register sources, transforms, destinations and post-processors in a job definition. It is important to understand that you can use Ruby logic at the DSL parsing time. This means that such code is possible, provided the CSV files are available at parsing time:
 
 ```ruby
-Dir['to_be_processed/*.csv'].each do |f|
+Dir['to_be_processed/*.csv'].each do |file|
   source MyCsvSource, file
 end
 ```
@@ -191,14 +200,32 @@ class MyCsvDestination
 end
 ```
 
-## Implementing post-processors
+## Implementing pre and post-processors
 
-Post-processors are currently blocks, which get called once, after the ETL
-successfully processed all the rows. It won't get called if an error occurred.
+Pre-processors and post-processors are currently blocks, which get called only once per ETL run:
+- Pre-processors get called before the ETL starts reading rows from the sources.
+- Post-processors get invoked after the ETL successfully processed all the rows.
+
+Note that post-processors won't get called if an error occurred earlier.
 
 ```ruby
 count = 0
 
+def system!(cmd)
+  fail "Command #{cmd} failed" unless system(cmd)
+end
+
+file = 'my_file.csv'
+sample_file = 'my_file.sample.csv'
+
+pre_process do
+  # it's handy to work with a reduced data set. you can
+  # e.g. just keep one line of the CSV files + the headers
+  system! "sed -n \"1p;25706p\" #{file} > #{sample_file}"
+end
+
+source MyCsv, file: sample_file
+
 transform do |row|
   count += 1
   row
@@ -225,6 +252,10 @@ The ability to support that DSL, but also check command line arguments, environm
 
 Make sure to subscribe to my [Ruby ETL blog](http://thibautbarrere.com) where I'll demonstrate such techniques over time!
 
+## Supported Ruby versions
+
+Kiba currently supports Ruby 2.0+ and JRuby (with its default 1.9 syntax).
+
 ## History & Credits
 
 Wow, you're still there? Nice to meet you. I'm [Thibaut](http://thibautbarrere.com), author of Kiba.

data/Rakefile

@@ -4,4 +4,4 @@ Rake::TestTask.new(:test) do |t|
   t.pattern = 'test/test_*.rb'
 end
 
-task :default => :test
+task default: :test

data/bin/kiba

@@ -2,4 +2,4 @@
 
 require_relative '../lib/kiba/cli'
 
-Kiba::Cli.run(ARGV)
+Kiba::Cli.run(ARGV)

data/kiba.gemspec

@@ -2,19 +2,19 @@
 require File.expand_path('../lib/kiba/version', __FILE__)
 
 Gem::Specification.new do |gem|
-  gem.authors       = ["Thibaut Barrère"]
-  gem.email         = ["thibaut.barrere@gmail.com"]
-  gem.description   = gem.summary = "Lightweight ETL for Ruby"
-  gem.homepage      = "http://thbar.github.io/kiba/"
-  gem.license       = "LGPL-3.0"
+  gem.authors       = ['Thibaut Barrère']
+  gem.email         = ['thibaut.barrere@gmail.com']
+  gem.description   = gem.summary = 'Lightweight ETL for Ruby'
+  gem.homepage      = 'http://thbar.github.io/kiba/'
+  gem.license       = 'LGPL-3.0'
   gem.files         = `git ls-files | grep -Ev '^(examples)'`.split("\n")
   gem.test_files    = `git ls-files -- test/*`.split("\n")
-  gem.name          = "kiba"
-  gem.require_paths = ["lib"]
+  gem.name          = 'kiba'
+  gem.require_paths = ['lib']
   gem.version       = Kiba::VERSION
   gem.executables   = ['kiba']
-  
+
   gem.add_development_dependency 'rake'
   gem.add_development_dependency 'minitest'
   gem.add_development_dependency 'awesome_print'
-end
+end

data/lib/kiba/cli.rb

@@ -4,8 +4,8 @@ module Kiba
   class Cli
     def self.run(args)
       unless args.size == 1
-        puts "Syntax: kiba your-script.etl"
-        exit -1
+        puts 'Syntax: kiba your-script.etl'
+        exit(-1)
       end
       filename = args[0]
       script_content = IO.read(filename)
@@ -13,4 +13,4 @@ module Kiba
       Kiba.run(job_definition)
     end
   end
-end
+end

data/lib/kiba/context.rb

@@ -5,24 +5,28 @@ module Kiba
       @control = control
     end
 
+    def pre_process(&block)
+      @control.pre_processes << block
+    end
+
     def source(klass, *initialization_params)
-      @control.sources << {klass: klass, args: initialization_params}
+      @control.sources << { klass: klass, args: initialization_params }
     end
 
     def transform(klass = nil, *initialization_params, &block)
       if klass
-        @control.transforms << {klass: klass, args: initialization_params}
+        @control.transforms << { klass: klass, args: initialization_params }
       else
         @control.transforms << block
       end
     end
 
     def destination(klass, *initialization_params)
-      @control.destinations << {klass: klass, args: initialization_params}
+      @control.destinations << { klass: klass, args: initialization_params }
     end
 
     def post_process(&block)
       @control.post_processes << block
     end
   end
-end
+end

data/lib/kiba/control.rb

@@ -1,5 +1,9 @@
 module Kiba
   class Control
+    def pre_processes
+      @pre_processes ||= []
+    end
+
     def sources
       @sources ||= []
     end
@@ -16,4 +20,4 @@ module Kiba
       @post_processes ||= []
     end
   end
-end
+end

data/lib/kiba/parser.rb

@@ -12,4 +12,4 @@ module Kiba
       control
     end
   end
-end
+end

data/lib/kiba/runner.rb

@@ -1,15 +1,25 @@
 module Kiba
   module Runner
     def run(control)
+      # instantiate early so that error are raised before any processing occurs
+      pre_processes = to_instances(control.pre_processes, true, false)
       sources = to_instances(control.sources)
       destinations = to_instances(control.destinations)
       transforms = to_instances(control.transforms, true)
-      # not using keyword args because JRuby defaults to 1.9 syntax currently
       post_processes = to_instances(control.post_processes, true, false)
 
+      pre_processes.each(&:call)
+      process_rows(sources, transforms, destinations)
+      destinations.each(&:close)
+      post_processes.each(&:call)
+    end
+    
+    def process_rows(sources, transforms, destinations)
       sources.each do |source|
         source.each do |row|
-          transforms.each_with_index do |transform, index|
+          transforms.each do |transform|
+            # TODO: avoid the case completely by e.g. subclassing Proc
+            # and aliasing `process` to `call`. Benchmark needed first though.
             if transform.is_a?(Proc)
               row = transform.call(row)
             else
@@ -23,22 +33,20 @@ module Kiba
           end
         end
       end
-
-      destinations.each(&:close)
-      post_processes.each(&:call)
     end
 
+    # not using keyword args because JRuby defaults to 1.9 syntax currently
     def to_instances(definitions, allow_block = false, allow_class = true)
       definitions.map do |d|
         case d
         when Proc
-          raise "Block form is not allowed here" unless allow_block
+          fail 'Block form is not allowed here' unless allow_block
           d
         else
-          raise "Class form is not allowed here" unless allow_class
+          fail 'Class form is not allowed here' unless allow_class
           d[:klass].new(*d[:args])
         end
       end
     end
   end
-end
+end

data/lib/kiba/version.rb

@@ -1,3 +1,3 @@
 module Kiba
-  VERSION = "0.5.0"
-end
+  VERSION = '0.6.0'
+end

data/test/helper.rb

@@ -7,7 +7,7 @@ class Kiba::Test < Minitest::Test
 
   def remove_files(*files)
     files.each do |file|
-      File.delete(file) if File.exists?(file)
+      File.delete(file) if File.exist?(file)
     end
   end
 

data/test/support/test_enumerable_source.rb

@@ -8,4 +8,4 @@ class TestEnumerableSource
       yield row
     end
   end
-end
+end

data/test/test_cli.rb

@@ -11,7 +11,7 @@ class TestCli < Kiba::Test
       Kiba::Cli.run([fixture('bogus.etl')])
     end
 
-    assert_match /uninitialized constant (.*)UnknownThing/, exception.message
+    assert_match(/uninitialized constant (.*)UnknownThing/, exception.message)
     assert_includes exception.backtrace.to_s, 'test/fixtures/bogus.etl:2:in'
   end
-end
+end

data/test/test_integration.rb

@@ -9,7 +9,8 @@ class TestIntegration < Kiba::Test
   let(:output_file) { 'test/tmp/output.csv' }
   let(:input_file) { 'test/tmp/input.csv' }
 
-  let(:sample_csv_data) do <<CSV
+  let(:sample_csv_data) do
+    <<CSV
 first_name,last_name,sex
 John,Doe,M
 Mary,Johnson,F
@@ -26,17 +27,17 @@ CSV
   def teardown
     remove_files(input_file, output_file)
   end
-  
+
   def test_csv_to_csv
-    # parse the ETL script (this won't run it)  
+    # parse the ETL script (this won't run it)
     control = Kiba.parse do
       source TestCsvSource, 'test/tmp/input.csv'
 
       transform do |row|
         row[:sex] = case row[:sex]
-        when 'M'; 'Male'
-        when 'F'; 'Female'
-        else 'Unknown'
+                    when 'M' then 'Male'
+                    when 'F' then 'Female'
+                    else 'Unknown'
         end
         row # must be returned
       end
@@ -61,28 +62,35 @@ Mary,Johnson,Female
 Cindy,Backgammon,Female
 CSV
   end
-  
+
   def test_variable_access
     message = nil
-    
+
     control = Kiba.parse do
       source TestEnumerableSource, [1, 2, 3]
-      
+
+      # assign a first value at parsing time
       count = 0
       
+      pre_process do
+        # then change it from there (run time)
+        count += 100
+      end
+
       transform do |r|
+        # increase it once per row
         count += 1
         r
       end
-      
+
       post_process do
-        message = "#{count} rows processed"
+        # and save so we can assert
+        message = "Count is now #{count}"
       end
     end
-    
+
     Kiba.run(control)
-    
-    assert_equal '3 rows processed', message
+
+    assert_equal 'Count is now 103', message
   end
-  
-end
+end

data/test/test_parser.rb

@@ -10,11 +10,11 @@ class TestParser < Kiba::Test
     control = Kiba.parse do
       source DummyClass, 'has', 'args'
     end
-    
+
     assert_equal DummyClass, control.sources[0][:klass]
-    assert_equal ['has', 'args'], control.sources[0][:args]
+    assert_equal %w(has args), control.sources[0][:args]
   end
-  
+
   def test_block_transform_definition
     control = Kiba.parse do
       transform { |row| row }
@@ -31,34 +31,42 @@ class TestParser < Kiba::Test
     assert_equal TestRenameFieldTransform, control.transforms[0][:klass]
     assert_equal [:last_name, :name], control.transforms[0][:args]
   end
-  
+
   def test_destination_definition
     control = Kiba.parse do
       destination DummyClass, 'has', 'args'
     end
-    
+
     assert_equal DummyClass, control.destinations[0][:klass]
-    assert_equal ['has', 'args'], control.destinations[0][:args]
+    assert_equal %w(has args), control.destinations[0][:args]
   end
-  
+
   def test_block_post_process_definition
     control = Kiba.parse do
-      post_process { }
+      post_process {}
     end
-    
+
     assert_instance_of Proc, control.post_processes[0]
   end
   
+  def test_block_pre_process_definition
+    control = Kiba.parse do
+      pre_process {}
+    end
+
+    assert_instance_of Proc, control.pre_processes[0]
+  end
+
   def test_source_as_string_parsing
     control = Kiba.parse <<RUBY
       source DummyClass, 'from', 'file'
 RUBY
-    
+
     assert_equal 1, control.sources.size
     assert_equal DummyClass, control.sources[0][:klass]
-    assert_equal ['from', 'file'], control.sources[0][:args]
+    assert_equal %w(from file), control.sources[0][:args]
   end
-  
+
   def test_source_as_file_doing_require
     IO.write 'test/tmp/etl-common.rb', <<RUBY
       def common_source_declaration
@@ -67,18 +75,18 @@ RUBY
 RUBY
     IO.write 'test/tmp/etl-main.rb', <<RUBY
       require './test/tmp/etl-common.rb'
-      
+
       source DummyClass, 'from', 'main'
       common_source_declaration
 RUBY
     control = Kiba.parse IO.read('test/tmp/etl-main.rb')
-    
+
     assert_equal 2, control.sources.size
 
-    assert_equal ['from', 'main'], control.sources[0][:args]
-    assert_equal ['from', 'common'], control.sources[1][:args]
-    
+    assert_equal %w(from main), control.sources[0][:args]
+    assert_equal %w(from common), control.sources[1][:args]
+
   ensure
     remove_files('test/tmp/etl-common.rb', 'test/tmp/etl-main.rb')
   end
-end
+end

data/test/test_runner.rb

@@ -3,10 +3,20 @@ require_relative 'helper'
 require_relative 'support/test_enumerable_source'
 
 class TestRunner < Kiba::Test
+  let(:rows) do
+    [
+      { field: 'value' },
+      { field: 'other-value' }
+    ]
+  end
+  
   let(:control) do
     control = Kiba::Control.new
     # this will yield a single row for testing
-    control.sources << {klass: TestEnumerableSource, args: [[{field: 'value'}]]} 
+    control.sources << {
+      klass: TestEnumerableSource,
+      args: [rows]
+    }
     control
   end
 
@@ -18,23 +28,32 @@ class TestRunner < Kiba::Test
   end
 
   def test_dismissed_row_not_passed_to_next_transform
-    control.transforms << lambda { |r| nil }
-    control.transforms << lambda { |r| @called = true; nil}
+    control.transforms << lambda { |_| nil }
+    control.transforms << lambda { |_| @called = true; nil }
     Kiba.run(control)
     assert_nil @called
   end
-  
-  def test_post_process_runs
-    control.post_processes << lambda { @called = true }
+
+  def test_post_process_runs_once
+    assert_equal 2, rows.size
+    @called = 0
+    control.post_processes << lambda { @called += 1 }
     Kiba.run(control)
-    assert_equal true, @called
+    assert_equal 1, @called
   end
-  
+
   def test_post_process_not_called_after_row_failure
-    control.transforms << lambda { |r| raise 'FAIL' }
+    control.transforms << lambda { |_| fail 'FAIL' }
     control.post_processes << lambda { @called = true }
     assert_raises(RuntimeError, 'FAIL') { Kiba.run(control) }
     assert_nil @called
   end
-
-end
+  
+  def test_pre_process_runs_once
+    assert_equal 2, rows.size
+    @called = 0
+    control.pre_processes << lambda { @called += 1 }
+    Kiba.run(control)
+    assert_equal 1, @called
+  end
+end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: kiba
 version: !ruby/object:Gem::Version
-  version: 0.5.0
+  version: 0.6.0
 platform: ruby
 authors:
 - Thibaut Barrère
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2015-04-18 00:00:00.000000000 Z
+date: 2015-05-14 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rake