metacrunch 3.0.3 → 3.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 308e14dd60e62be0591e1759b55e38e4441a3617
-  data.tar.gz: 58f330b33c6dc05f5997d88f908eb491e99b0929
+  metadata.gz: d90bf4e6defeb611ac43d6580734530b536b720c
+  data.tar.gz: bbabcd407de1dbad2a36f6a9fec9aa1a9f3a1fc5
 SHA512:
-  metadata.gz: dd28f9e417ec5a4a82a41bb7be0a32a5648ef9fbf7fa7434a978fb672edb112d5de3fa87b062631c2337c9d79c473215e247ed15fe708930a9ef81aaa25cc4a6
-  data.tar.gz: bbcaa2708128e6b5bb2fddfcf85060b8ca352579c2de0ae5e2fb7650e9af1f3313bea231b9a0309b5f19c9055ee80964a186411f86fe17731b3b1fae5147fd7a
+  metadata.gz: 330a3a2d2dd05198b60b4595c712a90baf64c725116492926c163903a276f9389ebaa4803e96489392a8a8b659a2cb8c1982d626dc2a0555a98e8c4887d9b96e
+  data.tar.gz: 151f47915a0bf4527f6b027ca241f3046d9ea01ef94d70fabcc4c8aa3b17dfd1b79871fbabff0e7db6ab58ca9737ec7aeb3e3075a28df6b784f90199db59f21a

data/.travis.yml

@@ -1,4 +1,4 @@
 language: ruby
 rvm:
-  - ruby-2.3.0
+  - ruby-2.3.1
   - jruby-9.0.5.0

data/Readme.md

@@ -8,6 +8,7 @@ metacrunch
 metacrunch is a simple and lightweight data processing and ETL ([Extract-Transform-Load](http://en.wikipedia.org/wiki/Extract,_transform,_load))
 toolkit for Ruby.
 
+**NOTE: THIS README IS FOR THE MASTER BRANCH. CHECK THE [RELEASES-PAGE](https://github.com/ubpb/metacrunch/releases) TO SEE THE README FOR THE RELEVANT RELEASES**
 
 Installation
 ------------
@@ -22,13 +23,14 @@ Creating ETL jobs
 
 The basic idea behind an ETL job in metacrunch is the concept of a data processing pipeline. Each ETL job reads data from one or more **sources** (extract step), runs one or more **transformations** (transform step) on the data and finally writes the transformed data to one or more **destinations** (load step).
 
-metacrunch provides you with a simple DSL to define and run such ETL jobs. Just create a text file with the extension `.metacrunch`. *Note: The extension doesn't really matter but you should avoid `.rb` to not loading them by mistake from another Ruby component.*
+metacrunch provides you with a simple DSL to define and run such ETL jobs in Ruby. Just create a text file with the extension `.metacrunch` and [run it](#running-etl-jobs) with the provided `metacrunch` CLI command. *Note: The extension doesn't really matter but you should avoid `.rb` to not loading them by mistake from another Ruby component.*
 
 Let's walk through the main steps of creating ETL jobs with metacrunch. For a collection of working examples check out our [metacrunch-demo](https://github.com/ubpb/metacrunch-demo) repo.
 
 #### It's Ruby
 
-Every `.metacrunch` job file is a regular Ruby file. So you can always use regular stuff like e.g. declaring methods, classes, variable and requiring other Ruby files. 
+Every `.metacrunch` job is a regular Ruby file and you can use any valid Ruby code like declaring methods, classes, variables, requiring other Ruby 
+files and so on. 
 
 ```ruby
 # File: my_etl_job.metacrunch
@@ -130,7 +132,40 @@ post_process MyCallable.new
 
 #### Defining options
 
-TBD.
+metacrunch has build-in support to parameterize your jobs. Using the `option` helper, you can declare options that can be set/overridden by the CLI when [running your jobs](#running-etl-jobs). 
+
+```ruby
+options do
+  add :number_of_processes, "-n", "--no-of-processes N", "Number of processes", default: 2 
+  add :database_url, "-d", "--database URL", "Database connection URL", required: true
+end
+```
+
+In this example we declare two options `number_of_processes` and `database_url`. `number_of_processes` defaults to 2, whereas `database_url` has no default and is required. In your job file you can access the option values using the `options` Hash. E.g. `options[:number_of_processes]`.
+
+To set/override these options use the command line.
+
+```
+$ bundle exec metacrunch my_etl_job.metacrunch @@ --no-of-processes 4
+```
+
+This will set the `options[:number_of_processes]` to `4`.
+
+To get a list of available options for a job, use `--help` on the command line.
+
+```
+$ bundle exec metacrunch my_etl_job.metacrunch @@ --help
+
+Usage: metacrunch run [options] JOB_FILE @@ [job-options] [ARGS]
+Job options:
+    -n, --no-of-processes N          Number of processes
+                                     DEFAULT: 2
+    -d, --database URL               Database connection URL
+                                     REQUIRED
+```
+
+To learn more about defining options take a look at the [reference below](#defining-job-options).
+
 
 Running ETL jobs
 ----------------
@@ -139,7 +174,7 @@ metacrunch comes with a handy command line tool. In a terminal use
 
 
 ```
-$ metacrunch run my_etl_job.metacrunch
+$ metacrunch my_etl_job.metacrunch
 ```
 
 to run a job.
@@ -147,7 +182,7 @@ to run a job.
 If you use [Bundler](http://bundler.io) to manage dependencies for your jobs make sure to change into the directory where your Gemfile is (or set BUNDLE_GEMFILE environment variable) and run metacrunch with `bundle exec`.
 
 ```
-$ bundle exec metacrunch run my_etl_job.metacrunch
+$ bundle exec metacrunch my_etl_job.metacrunch
 ```
 
 Depending on your environment `bundle exec` may not be required (e.g. you have rubygems-bundler installed) but we recommend using it whenever you have a Gemfile you like to use. When using Bundler make sure to add `gem "metacrunch"` to the Gemfile.
@@ -157,7 +192,7 @@ To pass options to the job, separate job options from the metacrunch command opt
 Use the following syntax
 
 ```
-$ [bundle exec] metacrunch run [COMMAND_OPTIONS] JOB_FILE [@@ [JOB_OPTIONS] [JOB_ARGS...]]
+$ [bundle exec] metacrunch [COMMAND_OPTIONS] JOB_FILE [@@ [JOB_OPTIONS] [JOB_ARGS...]]
 ```
 
 

data/lib/metacrunch/cli.rb

@@ -1,60 +1,110 @@
+require "optparse"
+
 module Metacrunch
   class Cli
     ARGS_SEPERATOR = "@@"
 
     def run
-      init_commander!
-      init_run_command!
-      run_commander!
+      job_files = global_parser.parse!(global_argv)
+
+      run!(job_files)
     end
 
   private
-    def commander
-      @commander ||= Commander::Runner.new(metacrunch_args)
-    end
+    def global_parser
+      @global_parser ||= OptionParser.new do |opts|
+        opts.banner = <<-BANNER.strip_heredoc
+          #{ColorizedString["Usage:"].bold}
+
+            metacrunch [options] JOB_FILE @@ [job-options] [ARGS...]
+
+          #{ColorizedString["Options:"].bold}
+        BANNER
+
+        opts.on("-v", "--version", "Show metacrunch version and exit") do
+          show_version
+        end
 
-    def init_commander!
-      commander.program :name, "metacrunch"
-      commander.program :version, Metacrunch::VERSION
-      commander.program :description, "Data processing and ETL toolkit for Ruby."
-      commander.default_command :help
+        opts.on("-n INTEGER", "--number-of-processes INTEGER", Integer, "Number of parallel processes to run the job. Source needs to support this. DEFAULT: 1") do |n|
+          error("--number-of-procs must be > 0") if n <= 0
+          global_options[:number_of_processes] = n
+        end
+
+        opts.separator "\n"
+      end
     end
 
-    def run_commander!
-      commander.run!
+    def global_options
+      @global_options ||= {
+        number_of_processes: 1
+      }
     end
 
-    def init_run_command!
-      commander.command :run do |c|
-        c.syntax = "metacrunch run [options] FILE [@@ job_options]"
-        c.description = "Runs a metacrunch job description."
+    def show_version
+      puts Metacrunch::VERSION
+      exit(0)
+    end
 
-        c.action do |filenames, program_options|
-          if filenames.empty?
-            say "You need to provide a job description file."
-            exit(1)
-          elsif filenames.count > 1
-            say "You must provide exactly one job description file."
-          else
-            filename = File.expand_path(filenames.first)
-            dir = File.dirname(filename)
+    def error(message)
+      puts ColorizedString["Error: #{message}\n"].red.bold
+      puts global_parser.help
+      exit(0)
+    end
 
-            Dir.chdir(dir) do
-              Metacrunch::Job.define(File.read(filename), filename: filename, args: job_args).run
-            end
-          end
-        end
+    def global_argv
+      index = ARGV.index(ARGS_SEPERATOR)
+      if index == 0
+        []
+      else
+        @global_argv ||= index ? ARGV[0..index-1] : ARGV
       end
     end
 
-    def metacrunch_args
+    def job_argv
       index = ARGV.index(ARGS_SEPERATOR)
-      @metacrunch_args ||= index ? ARGV[0..index-1] : ARGV
+      @job_argv ||= index ? ARGV[index+1..-1] : nil
     end
 
-    def job_args
-      index = ARGV.index(ARGS_SEPERATOR)
-      @job_args ||= index ? ARGV[index+1..-1] : nil
+    def run!(job_files)
+      if job_files.first == "run"
+        puts ColorizedString["WARN: Using 'run' is deprecated. Just use 'metacrunch [options] JOB_FILE @@ [job-options] [ARGS...]'\n"].yellow.bold
+        job_files = job_files[1..-1]
+      end
+
+      if job_files.empty?
+        error "You need to provide a job file."
+      elsif job_files.count > 1
+        error "You must provide exactly one job file."
+      else
+        job_filename = File.expand_path(job_files.first)
+        dir = File.dirname(job_filename)
+
+        Dir.chdir(dir) do
+          run_job!(job_filename)
+        end
+      end
+    end
+
+    def run_job!(job_filename)
+      if global_options[:number_of_processes] > 1
+        process_indicies = (0..(global_options[:number_of_processes] - 1)).to_a
+
+        Parallel.each(process_indicies) do |process_index|
+          Metacrunch::Job.define(
+            File.read(job_filename),
+            filename: job_filename,
+            args: job_argv,
+            number_of_processes: global_options[:number_of_processes],
+            process_index: process_index
+          ).run
+        end
+      else
+        Metacrunch::Job.define(
+          File.read(job_filename),
+          filename: job_filename,
+          args: job_argv
+        ).run
+      end
     end
 
   end

data/lib/metacrunch/db/reader.rb

@@ -1,5 +1,8 @@
+require "metacrunch/db"
+
 module Metacrunch
   class Db::Reader
+    include Metacrunch::ParallelProcessableReader
 
     def initialize(database_connection_or_url, dataset_proc, options = {})
       @rows_per_fetch = options.delete(:rows_per_fetch) || 1000
@@ -10,14 +13,29 @@ module Metacrunch
         database_connection_or_url
       end
 
-      @dataset = dataset_proc.call(@db)
+      @dataset = dataset_proc.call(@db).unlimited
+      @total_numbers_of_records = @dataset.count
+
+      unless @dataset.opts[:order]
+        raise ArgumentError, "Metacrunch::Db::Reader requires the dataset be ordered."
+      end
     end
 
     def each(&block)
       return enum_for(__method__) unless block_given?
 
-      @dataset.paged_each(rows_per_fetch: @rows_per_fetch) do |row|
-        yield(row)
+      @db.transaction do
+        offset = (-number_of_processes * @rows_per_fetch) + (process_index * @rows_per_fetch)
+
+        loop do
+          offset = offset + (number_of_processes * @rows_per_fetch)
+
+          @dataset.limit(@rows_per_fetch).offset(offset).each do |row|
+            yield(row)
+          end
+
+          break if offset + @rows_per_fetch >= @total_numbers_of_records
+        end
       end
 
       self

data/lib/metacrunch/db/writer.rb

@@ -1,9 +1,13 @@
+require "metacrunch/db"
+
 module Metacrunch
   class Db::Writer
 
     def initialize(database_connection_or_url, dataset_proc, options = {})
-      @use_upsert = options.delete(:use_upsert) || false
-      @id_key     = options.delete(:id_key)     || :id
+      @use_upsert          = options.delete(:use_upsert)          || false
+      @id_key              = options.delete(:id_key)              || :id
+      @isolation_level     = options.delete(:isolation_level)     || :repeatable
+      @transaction_retries = options.delete(:transaction_retries) || 5
 
       @db = if database_connection_or_url.is_a?(String)
         Sequel.connect(database_connection_or_url, options)
@@ -16,7 +20,7 @@ module Metacrunch
 
     def write(data)
       if data.is_a?(Array)
-        @db.transaction do
+        @db.transaction(isolation: @isolation_level, num_retries: @transaction_retries) do
           data.each{|d| insert_or_upsert(d) }
         end
       else

data/lib/metacrunch/job/dsl/option_support.rb

@@ -36,7 +36,7 @@ module Metacrunch
 
     def parser
       @parser ||= OptionParser.new do |parser|
-        parser.banner = "Usage: metacrunch run [options] JOB_FILE @@ [job-options] [ARGS]\nJob options:"
+        parser.banner = "Usage: metacrunch [options] JOB_FILE @@ [job-options] [ARGS]\nJob options:"
       end
     end
 

data/lib/metacrunch/job.rb

@@ -6,14 +6,16 @@ module Metacrunch
     attr_reader :builder, :args
 
     class << self
-      def define(file_content = nil, filename: nil, args: nil, &block)
-        self.new(file_content, filename: filename, args: args, &block)
+      def define(file_content = nil, filename: nil, args: nil, number_of_processes: 1, process_index: 0, &block)
+        self.new(file_content, filename: filename, args: args, number_of_processes: number_of_processes, process_index: process_index, &block)
       end
     end
 
-    def initialize(file_content = nil, filename: nil, args: nil, &block)
+    def initialize(file_content = nil, filename: nil, args: nil, number_of_processes: 1, process_index: 0, &block)
       @builder = Dsl.new(self)
       @args = args
+      @number_of_processes = number_of_processes
+      @process_index = process_index
 
       if file_content
         @builder.instance_eval(file_content, filename || "")
@@ -109,6 +111,18 @@ module Metacrunch
 
     def run_transformations
       sources.each do |source|
+        # Setup parallel processing
+        if @number_of_processes > 1
+          if source.class.included_modules.include?(Metacrunch::ParallelProcessableReader)
+            source.set_parallel_process_options(
+              number_of_processes: @number_of_processes,
+              process_index: @process_index
+            )
+          else
+            raise RuntimeError, "source does't support parallel processing"
+          end
+        end
+
         # sources are expected to respond to `each`
         source.each do |data|
           run_transformations_and_write_destinations(data)

data/lib/metacrunch/parallel_processable_reader.rb

@@ -0,0 +1,21 @@
+module Metacrunch
+  module ParallelProcessableReader
+
+    def set_parallel_process_options(number_of_processes: 1, process_index: 0)
+      raise ArgumentError, "number_of_processes must be >= 1" if number_of_processes < 1
+      raise ArgumentError, "process_index must be >= 0" if process_index < 0
+
+      @number_of_processes = number_of_processes
+      @process_index = process_index
+    end
+
+    def number_of_processes
+      @number_of_processes || 1
+    end
+
+    def process_index
+      @process_index || 0
+    end
+
+  end
+end

data/lib/metacrunch/redis/queue_reader.rb

@@ -0,0 +1,37 @@
+require "metacrunch/redis"
+
+module Metacrunch
+  class Redis::QueueReader
+
+    def initialize(redis_connection_or_url, queue_name, options = {})
+      @queue_name = queue_name
+      raise ArgumentError, "queue_name must be a string" unless queue_name.is_a?(String)
+
+      @blocking_mode = options.delete(:blocking) || false
+
+      @redis = if redis_connection_or_url.is_a?(String)
+        ::Redis.new(url: redis_connection_or_url)
+      else
+        redis_connection_or_url
+      end
+    end
+
+    def each(&block)
+      return enum_for(__method__) unless block_given?
+
+      if @blocking_mode
+        while true
+          result = @redis.blpop(@queue_name)
+          yield JSON.parse(result[1]) if result
+        end
+      else
+        while result = @redis.lpop(@queue_name)
+          yield JSON.parse(result)
+        end
+      end
+
+      self
+    end
+
+  end
+end

data/lib/metacrunch/redis/queue_writer.rb

@@ -0,0 +1,26 @@
+require "metacrunch/redis"
+
+module Metacrunch
+  class Redis::QueueWriter
+
+    def initialize(redis_connection_or_url, queue_name, options = {})
+      @queue_name = queue_name
+      raise ArgumentError, "queue_name must be a string" unless queue_name.is_a?(String)
+
+      @redis = if redis_connection_or_url.is_a?(String)
+        ::Redis.new(url: redis_connection_or_url)
+      else
+        redis_connection_or_url
+      end
+    end
+
+    def write(data)
+      @redis.rpush(@queue_name, data.to_json)
+    end
+
+    def close
+      @redis.close if @redis
+    end
+
+  end
+end

data/lib/metacrunch/redis.rb

@@ -0,0 +1,8 @@
+require "redis"
+
+module Metacrunch
+  class Redis
+    require_relative "redis/queue_reader"
+    require_relative "redis/queue_writer"
+  end
+end

data/lib/metacrunch/version.rb

@@ -1,3 +1,3 @@
 module Metacrunch
-  VERSION = "3.0.3"
+  VERSION = "3.1.0"
 end

data/lib/metacrunch.rb

@@ -1,12 +1,14 @@
 require "active_support"
 require "active_support/core_ext"
-require "commander"
-require "sequel"
+require "colorized_string"
+require "parallel"
 
 module Metacrunch
   require_relative "metacrunch/version"
   require_relative "metacrunch/cli"
   require_relative "metacrunch/job"
+  require_relative "metacrunch/parallel_processable_reader"
   require_relative "metacrunch/fs"
   require_relative "metacrunch/db"
+  require_relative "metacrunch/redis"
 end

data/metacrunch.gemspec

@@ -17,7 +17,9 @@ Gem::Specification.new do |spec|
   spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
   spec.require_paths = ["lib"]
 
-  spec.add_dependency "activesupport", ">= 4.2"
-  spec.add_dependency "commander",     "~> 4.4"
+  spec.add_dependency "activesupport", ">= 4.2", "< 5.1"
+  spec.add_dependency "colorize",      ">= 0.8"
+  spec.add_dependency "parallel",      "~> 1.9"
   spec.add_dependency "sequel",        "~> 4.33"
+  spec.add_dependency "redis",         "~> 3.3"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: metacrunch
 version: !ruby/object:Gem::Version
-  version: 3.0.3
+  version: 3.1.0
 platform: ruby
 authors:
 - René Sprotte
@@ -10,7 +10,7 @@ authors:
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2016-07-17 00:00:00.000000000 Z
+date: 2016-07-21 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport
@@ -19,6 +19,9 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '4.2'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '5.1'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
@@ -26,20 +29,37 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '4.2'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '5.1'
+- !ruby/object:Gem::Dependency
+  name: colorize
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0.8'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0.8'
 - !ruby/object:Gem::Dependency
-  name: commander
+  name: parallel
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '4.4'
+        version: '1.9'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '4.4'
+        version: '1.9'
 - !ruby/object:Gem::Dependency
   name: sequel
   requirement: !ruby/object:Gem::Requirement
@@ -54,6 +74,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '4.33'
+- !ruby/object:Gem::Dependency
+  name: redis
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.3'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.3'
 description: 
 email: r.sprotte@ub.uni-paderborn.de
 executables:
@@ -82,6 +116,10 @@ files:
 - lib/metacrunch/job/buffer.rb
 - lib/metacrunch/job/dsl.rb
 - lib/metacrunch/job/dsl/option_support.rb
+- lib/metacrunch/parallel_processable_reader.rb
+- lib/metacrunch/redis.rb
+- lib/metacrunch/redis/queue_reader.rb
+- lib/metacrunch/redis/queue_writer.rb
 - lib/metacrunch/test_utils.rb
 - lib/metacrunch/test_utils/dummy_callable.rb
 - lib/metacrunch/test_utils/dummy_destination.rb