metacrunch 3.0.3 → 3.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 308e14dd60e62be0591e1759b55e38e4441a3617
4
- data.tar.gz: 58f330b33c6dc05f5997d88f908eb491e99b0929
3
+ metadata.gz: d90bf4e6defeb611ac43d6580734530b536b720c
4
+ data.tar.gz: bbabcd407de1dbad2a36f6a9fec9aa1a9f3a1fc5
5
5
  SHA512:
6
- metadata.gz: dd28f9e417ec5a4a82a41bb7be0a32a5648ef9fbf7fa7434a978fb672edb112d5de3fa87b062631c2337c9d79c473215e247ed15fe708930a9ef81aaa25cc4a6
7
- data.tar.gz: bbcaa2708128e6b5bb2fddfcf85060b8ca352579c2de0ae5e2fb7650e9af1f3313bea231b9a0309b5f19c9055ee80964a186411f86fe17731b3b1fae5147fd7a
6
+ metadata.gz: 330a3a2d2dd05198b60b4595c712a90baf64c725116492926c163903a276f9389ebaa4803e96489392a8a8b659a2cb8c1982d626dc2a0555a98e8c4887d9b96e
7
+ data.tar.gz: 151f47915a0bf4527f6b027ca241f3046d9ea01ef94d70fabcc4c8aa3b17dfd1b79871fbabff0e7db6ab58ca9737ec7aeb3e3075a28df6b784f90199db59f21a
data/.travis.yml CHANGED
@@ -1,4 +1,4 @@
1
1
  language: ruby
2
2
  rvm:
3
- - ruby-2.3.0
3
+ - ruby-2.3.1
4
4
  - jruby-9.0.5.0
data/Readme.md CHANGED
@@ -8,6 +8,7 @@ metacrunch
8
8
  metacrunch is a simple and lightweight data processing and ETL ([Extract-Transform-Load](http://en.wikipedia.org/wiki/Extract,_transform,_load))
9
9
  toolkit for Ruby.
10
10
 
11
+ **NOTE: THIS README IS FOR THE MASTER BRANCH. CHECK THE [RELEASES-PAGE](https://github.com/ubpb/metacrunch/releases) TO SEE THE README FOR THE RELEVANT RELEASES**
11
12
 
12
13
  Installation
13
14
  ------------
@@ -22,13 +23,14 @@ Creating ETL jobs
22
23
 
23
24
  The basic idea behind an ETL job in metacrunch is the concept of a data processing pipeline. Each ETL job reads data from one or more **sources** (extract step), runs one or more **transformations** (transform step) on the data and finally writes the transformed data to one or more **destinations** (load step).
24
25
 
25
- metacrunch provides you with a simple DSL to define and run such ETL jobs. Just create a text file with the extension `.metacrunch`. *Note: The extension doesn't really matter but you should avoid `.rb` to not loading them by mistake from another Ruby component.*
26
+ metacrunch provides you with a simple DSL to define and run such ETL jobs in Ruby. Just create a text file with the extension `.metacrunch` and [run it](#running-etl-jobs) with the provided `metacrunch` CLI command. *Note: The extension doesn't really matter but you should avoid `.rb` to not loading them by mistake from another Ruby component.*
26
27
 
27
28
  Let's walk through the main steps of creating ETL jobs with metacrunch. For a collection of working examples check out our [metacrunch-demo](https://github.com/ubpb/metacrunch-demo) repo.
28
29
 
29
30
  #### It's Ruby
30
31
 
31
- Every `.metacrunch` job file is a regular Ruby file. So you can always use regular stuff like e.g. declaring methods, classes, variable and requiring other Ruby files.
32
+ Every `.metacrunch` job is a regular Ruby file and you can use any valid Ruby code like declaring methods, classes, variables, requiring other Ruby
33
+ files and so on.
32
34
 
33
35
  ```ruby
34
36
  # File: my_etl_job.metacrunch
@@ -130,7 +132,40 @@ post_process MyCallable.new
130
132
 
131
133
  #### Defining options
132
134
 
133
- TBD.
135
+ metacrunch has build-in support to parameterize your jobs. Using the `option` helper, you can declare options that can be set/overridden by the CLI when [running your jobs](#running-etl-jobs).
136
+
137
+ ```ruby
138
+ options do
139
+ add :number_of_processes, "-n", "--no-of-processes N", "Number of processes", default: 2
140
+ add :database_url, "-d", "--database URL", "Database connection URL", required: true
141
+ end
142
+ ```
143
+
144
+ In this example we declare two options `number_of_processes` and `database_url`. `number_of_processes` defaults to 2, whereas `database_url` has no default and is required. In your job file you can access the option values using the `options` Hash. E.g. `options[:number_of_processes]`.
145
+
146
+ To set/override these options use the command line.
147
+
148
+ ```
149
+ $ bundle exec metacrunch my_etl_job.metacrunch @@ --no-of-processes 4
150
+ ```
151
+
152
+ This will set the `options[:number_of_processes]` to `4`.
153
+
154
+ To get a list of available options for a job, use `--help` on the command line.
155
+
156
+ ```
157
+ $ bundle exec metacrunch my_etl_job.metacrunch @@ --help
158
+
159
+ Usage: metacrunch run [options] JOB_FILE @@ [job-options] [ARGS]
160
+ Job options:
161
+ -n, --no-of-processes N Number of processes
162
+ DEFAULT: 2
163
+ -d, --database URL Database connection URL
164
+ REQUIRED
165
+ ```
166
+
167
+ To learn more about defining options take a look at the [reference below](#defining-job-options).
168
+
134
169
 
135
170
  Running ETL jobs
136
171
  ----------------
@@ -139,7 +174,7 @@ metacrunch comes with a handy command line tool. In a terminal use
139
174
 
140
175
 
141
176
  ```
142
- $ metacrunch run my_etl_job.metacrunch
177
+ $ metacrunch my_etl_job.metacrunch
143
178
  ```
144
179
 
145
180
  to run a job.
@@ -147,7 +182,7 @@ to run a job.
147
182
  If you use [Bundler](http://bundler.io) to manage dependencies for your jobs make sure to change into the directory where your Gemfile is (or set BUNDLE_GEMFILE environment variable) and run metacrunch with `bundle exec`.
148
183
 
149
184
  ```
150
- $ bundle exec metacrunch run my_etl_job.metacrunch
185
+ $ bundle exec metacrunch my_etl_job.metacrunch
151
186
  ```
152
187
 
153
188
  Depending on your environment `bundle exec` may not be required (e.g. you have rubygems-bundler installed) but we recommend using it whenever you have a Gemfile you like to use. When using Bundler make sure to add `gem "metacrunch"` to the Gemfile.
@@ -157,7 +192,7 @@ To pass options to the job, separate job options from the metacrunch command opt
157
192
  Use the following syntax
158
193
 
159
194
  ```
160
- $ [bundle exec] metacrunch run [COMMAND_OPTIONS] JOB_FILE [@@ [JOB_OPTIONS] [JOB_ARGS...]]
195
+ $ [bundle exec] metacrunch [COMMAND_OPTIONS] JOB_FILE [@@ [JOB_OPTIONS] [JOB_ARGS...]]
161
196
  ```
162
197
 
163
198
 
@@ -1,60 +1,110 @@
1
+ require "optparse"
2
+
1
3
  module Metacrunch
2
4
  class Cli
3
5
  ARGS_SEPERATOR = "@@"
4
6
 
5
7
  def run
6
- init_commander!
7
- init_run_command!
8
- run_commander!
8
+ job_files = global_parser.parse!(global_argv)
9
+
10
+ run!(job_files)
9
11
  end
10
12
 
11
13
  private
12
- def commander
13
- @commander ||= Commander::Runner.new(metacrunch_args)
14
- end
14
+ def global_parser
15
+ @global_parser ||= OptionParser.new do |opts|
16
+ opts.banner = <<-BANNER.strip_heredoc
17
+ #{ColorizedString["Usage:"].bold}
18
+
19
+ metacrunch [options] JOB_FILE @@ [job-options] [ARGS...]
20
+
21
+ #{ColorizedString["Options:"].bold}
22
+ BANNER
23
+
24
+ opts.on("-v", "--version", "Show metacrunch version and exit") do
25
+ show_version
26
+ end
15
27
 
16
- def init_commander!
17
- commander.program :name, "metacrunch"
18
- commander.program :version, Metacrunch::VERSION
19
- commander.program :description, "Data processing and ETL toolkit for Ruby."
20
- commander.default_command :help
28
+ opts.on("-n INTEGER", "--number-of-processes INTEGER", Integer, "Number of parallel processes to run the job. Source needs to support this. DEFAULT: 1") do |n|
29
+ error("--number-of-procs must be > 0") if n <= 0
30
+ global_options[:number_of_processes] = n
31
+ end
32
+
33
+ opts.separator "\n"
34
+ end
21
35
  end
22
36
 
23
- def run_commander!
24
- commander.run!
37
+ def global_options
38
+ @global_options ||= {
39
+ number_of_processes: 1
40
+ }
25
41
  end
26
42
 
27
- def init_run_command!
28
- commander.command :run do |c|
29
- c.syntax = "metacrunch run [options] FILE [@@ job_options]"
30
- c.description = "Runs a metacrunch job description."
43
+ def show_version
44
+ puts Metacrunch::VERSION
45
+ exit(0)
46
+ end
31
47
 
32
- c.action do |filenames, program_options|
33
- if filenames.empty?
34
- say "You need to provide a job description file."
35
- exit(1)
36
- elsif filenames.count > 1
37
- say "You must provide exactly one job description file."
38
- else
39
- filename = File.expand_path(filenames.first)
40
- dir = File.dirname(filename)
48
+ def error(message)
49
+ puts ColorizedString["Error: #{message}\n"].red.bold
50
+ puts global_parser.help
51
+ exit(0)
52
+ end
41
53
 
42
- Dir.chdir(dir) do
43
- Metacrunch::Job.define(File.read(filename), filename: filename, args: job_args).run
44
- end
45
- end
46
- end
54
+ def global_argv
55
+ index = ARGV.index(ARGS_SEPERATOR)
56
+ if index == 0
57
+ []
58
+ else
59
+ @global_argv ||= index ? ARGV[0..index-1] : ARGV
47
60
  end
48
61
  end
49
62
 
50
- def metacrunch_args
63
+ def job_argv
51
64
  index = ARGV.index(ARGS_SEPERATOR)
52
- @metacrunch_args ||= index ? ARGV[0..index-1] : ARGV
65
+ @job_argv ||= index ? ARGV[index+1..-1] : nil
53
66
  end
54
67
 
55
- def job_args
56
- index = ARGV.index(ARGS_SEPERATOR)
57
- @job_args ||= index ? ARGV[index+1..-1] : nil
68
+ def run!(job_files)
69
+ if job_files.first == "run"
70
+ puts ColorizedString["WARN: Using 'run' is deprecated. Just use 'metacrunch [options] JOB_FILE @@ [job-options] [ARGS...]'\n"].yellow.bold
71
+ job_files = job_files[1..-1]
72
+ end
73
+
74
+ if job_files.empty?
75
+ error "You need to provide a job file."
76
+ elsif job_files.count > 1
77
+ error "You must provide exactly one job file."
78
+ else
79
+ job_filename = File.expand_path(job_files.first)
80
+ dir = File.dirname(job_filename)
81
+
82
+ Dir.chdir(dir) do
83
+ run_job!(job_filename)
84
+ end
85
+ end
86
+ end
87
+
88
+ def run_job!(job_filename)
89
+ if global_options[:number_of_processes] > 1
90
+ process_indicies = (0..(global_options[:number_of_processes] - 1)).to_a
91
+
92
+ Parallel.each(process_indicies) do |process_index|
93
+ Metacrunch::Job.define(
94
+ File.read(job_filename),
95
+ filename: job_filename,
96
+ args: job_argv,
97
+ number_of_processes: global_options[:number_of_processes],
98
+ process_index: process_index
99
+ ).run
100
+ end
101
+ else
102
+ Metacrunch::Job.define(
103
+ File.read(job_filename),
104
+ filename: job_filename,
105
+ args: job_argv
106
+ ).run
107
+ end
58
108
  end
59
109
 
60
110
  end
@@ -1,5 +1,8 @@
1
+ require "metacrunch/db"
2
+
1
3
  module Metacrunch
2
4
  class Db::Reader
5
+ include Metacrunch::ParallelProcessableReader
3
6
 
4
7
  def initialize(database_connection_or_url, dataset_proc, options = {})
5
8
  @rows_per_fetch = options.delete(:rows_per_fetch) || 1000
@@ -10,14 +13,29 @@ module Metacrunch
10
13
  database_connection_or_url
11
14
  end
12
15
 
13
- @dataset = dataset_proc.call(@db)
16
+ @dataset = dataset_proc.call(@db).unlimited
17
+ @total_numbers_of_records = @dataset.count
18
+
19
+ unless @dataset.opts[:order]
20
+ raise ArgumentError, "Metacrunch::Db::Reader requires the dataset be ordered."
21
+ end
14
22
  end
15
23
 
16
24
  def each(&block)
17
25
  return enum_for(__method__) unless block_given?
18
26
 
19
- @dataset.paged_each(rows_per_fetch: @rows_per_fetch) do |row|
20
- yield(row)
27
+ @db.transaction do
28
+ offset = (-number_of_processes * @rows_per_fetch) + (process_index * @rows_per_fetch)
29
+
30
+ loop do
31
+ offset = offset + (number_of_processes * @rows_per_fetch)
32
+
33
+ @dataset.limit(@rows_per_fetch).offset(offset).each do |row|
34
+ yield(row)
35
+ end
36
+
37
+ break if offset + @rows_per_fetch >= @total_numbers_of_records
38
+ end
21
39
  end
22
40
 
23
41
  self
@@ -1,9 +1,13 @@
1
+ require "metacrunch/db"
2
+
1
3
  module Metacrunch
2
4
  class Db::Writer
3
5
 
4
6
  def initialize(database_connection_or_url, dataset_proc, options = {})
5
- @use_upsert = options.delete(:use_upsert) || false
6
- @id_key = options.delete(:id_key) || :id
7
+ @use_upsert = options.delete(:use_upsert) || false
8
+ @id_key = options.delete(:id_key) || :id
9
+ @isolation_level = options.delete(:isolation_level) || :repeatable
10
+ @transaction_retries = options.delete(:transaction_retries) || 5
7
11
 
8
12
  @db = if database_connection_or_url.is_a?(String)
9
13
  Sequel.connect(database_connection_or_url, options)
@@ -16,7 +20,7 @@ module Metacrunch
16
20
 
17
21
  def write(data)
18
22
  if data.is_a?(Array)
19
- @db.transaction do
23
+ @db.transaction(isolation: @isolation_level, num_retries: @transaction_retries) do
20
24
  data.each{|d| insert_or_upsert(d) }
21
25
  end
22
26
  else
@@ -36,7 +36,7 @@ module Metacrunch
36
36
 
37
37
  def parser
38
38
  @parser ||= OptionParser.new do |parser|
39
- parser.banner = "Usage: metacrunch run [options] JOB_FILE @@ [job-options] [ARGS]\nJob options:"
39
+ parser.banner = "Usage: metacrunch [options] JOB_FILE @@ [job-options] [ARGS]\nJob options:"
40
40
  end
41
41
  end
42
42
 
@@ -6,14 +6,16 @@ module Metacrunch
6
6
  attr_reader :builder, :args
7
7
 
8
8
  class << self
9
- def define(file_content = nil, filename: nil, args: nil, &block)
10
- self.new(file_content, filename: filename, args: args, &block)
9
+ def define(file_content = nil, filename: nil, args: nil, number_of_processes: 1, process_index: 0, &block)
10
+ self.new(file_content, filename: filename, args: args, number_of_processes: number_of_processes, process_index: process_index, &block)
11
11
  end
12
12
  end
13
13
 
14
- def initialize(file_content = nil, filename: nil, args: nil, &block)
14
+ def initialize(file_content = nil, filename: nil, args: nil, number_of_processes: 1, process_index: 0, &block)
15
15
  @builder = Dsl.new(self)
16
16
  @args = args
17
+ @number_of_processes = number_of_processes
18
+ @process_index = process_index
17
19
 
18
20
  if file_content
19
21
  @builder.instance_eval(file_content, filename || "")
@@ -109,6 +111,18 @@ module Metacrunch
109
111
 
110
112
  def run_transformations
111
113
  sources.each do |source|
114
+ # Setup parallel processing
115
+ if @number_of_processes > 1
116
+ if source.class.included_modules.include?(Metacrunch::ParallelProcessableReader)
117
+ source.set_parallel_process_options(
118
+ number_of_processes: @number_of_processes,
119
+ process_index: @process_index
120
+ )
121
+ else
122
+ raise RuntimeError, "source does't support parallel processing"
123
+ end
124
+ end
125
+
112
126
  # sources are expected to respond to `each`
113
127
  source.each do |data|
114
128
  run_transformations_and_write_destinations(data)
@@ -0,0 +1,21 @@
1
+ module Metacrunch
2
+ module ParallelProcessableReader
3
+
4
+ def set_parallel_process_options(number_of_processes: 1, process_index: 0)
5
+ raise ArgumentError, "number_of_processes must be >= 1" if number_of_processes < 1
6
+ raise ArgumentError, "process_index must be >= 0" if process_index < 0
7
+
8
+ @number_of_processes = number_of_processes
9
+ @process_index = process_index
10
+ end
11
+
12
+ def number_of_processes
13
+ @number_of_processes || 1
14
+ end
15
+
16
+ def process_index
17
+ @process_index || 0
18
+ end
19
+
20
+ end
21
+ end
@@ -0,0 +1,37 @@
1
+ require "metacrunch/redis"
2
+
3
+ module Metacrunch
4
+ class Redis::QueueReader
5
+
6
+ def initialize(redis_connection_or_url, queue_name, options = {})
7
+ @queue_name = queue_name
8
+ raise ArgumentError, "queue_name must be a string" unless queue_name.is_a?(String)
9
+
10
+ @blocking_mode = options.delete(:blocking) || false
11
+
12
+ @redis = if redis_connection_or_url.is_a?(String)
13
+ ::Redis.new(url: redis_connection_or_url)
14
+ else
15
+ redis_connection_or_url
16
+ end
17
+ end
18
+
19
+ def each(&block)
20
+ return enum_for(__method__) unless block_given?
21
+
22
+ if @blocking_mode
23
+ while true
24
+ result = @redis.blpop(@queue_name)
25
+ yield JSON.parse(result[1]) if result
26
+ end
27
+ else
28
+ while result = @redis.lpop(@queue_name)
29
+ yield JSON.parse(result)
30
+ end
31
+ end
32
+
33
+ self
34
+ end
35
+
36
+ end
37
+ end
@@ -0,0 +1,26 @@
1
+ require "metacrunch/redis"
2
+
3
+ module Metacrunch
4
+ class Redis::QueueWriter
5
+
6
+ def initialize(redis_connection_or_url, queue_name, options = {})
7
+ @queue_name = queue_name
8
+ raise ArgumentError, "queue_name must be a string" unless queue_name.is_a?(String)
9
+
10
+ @redis = if redis_connection_or_url.is_a?(String)
11
+ ::Redis.new(url: redis_connection_or_url)
12
+ else
13
+ redis_connection_or_url
14
+ end
15
+ end
16
+
17
+ def write(data)
18
+ @redis.rpush(@queue_name, data.to_json)
19
+ end
20
+
21
+ def close
22
+ @redis.close if @redis
23
+ end
24
+
25
+ end
26
+ end
@@ -0,0 +1,8 @@
1
+ require "redis"
2
+
3
+ module Metacrunch
4
+ class Redis
5
+ require_relative "redis/queue_reader"
6
+ require_relative "redis/queue_writer"
7
+ end
8
+ end
@@ -1,3 +1,3 @@
1
1
  module Metacrunch
2
- VERSION = "3.0.3"
2
+ VERSION = "3.1.0"
3
3
  end
data/lib/metacrunch.rb CHANGED
@@ -1,12 +1,14 @@
1
1
  require "active_support"
2
2
  require "active_support/core_ext"
3
- require "commander"
4
- require "sequel"
3
+ require "colorized_string"
4
+ require "parallel"
5
5
 
6
6
  module Metacrunch
7
7
  require_relative "metacrunch/version"
8
8
  require_relative "metacrunch/cli"
9
9
  require_relative "metacrunch/job"
10
+ require_relative "metacrunch/parallel_processable_reader"
10
11
  require_relative "metacrunch/fs"
11
12
  require_relative "metacrunch/db"
13
+ require_relative "metacrunch/redis"
12
14
  end
data/metacrunch.gemspec CHANGED
@@ -17,7 +17,9 @@ Gem::Specification.new do |spec|
17
17
  spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
18
18
  spec.require_paths = ["lib"]
19
19
 
20
- spec.add_dependency "activesupport", ">= 4.2"
21
- spec.add_dependency "commander", "~> 4.4"
20
+ spec.add_dependency "activesupport", ">= 4.2", "< 5.1"
21
+ spec.add_dependency "colorize", ">= 0.8"
22
+ spec.add_dependency "parallel", "~> 1.9"
22
23
  spec.add_dependency "sequel", "~> 4.33"
24
+ spec.add_dependency "redis", "~> 3.3"
23
25
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: metacrunch
3
3
  version: !ruby/object:Gem::Version
4
- version: 3.0.3
4
+ version: 3.1.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - René Sprotte
@@ -10,7 +10,7 @@ authors:
10
10
  autorequire:
11
11
  bindir: exe
12
12
  cert_chain: []
13
- date: 2016-07-17 00:00:00.000000000 Z
13
+ date: 2016-07-21 00:00:00.000000000 Z
14
14
  dependencies:
15
15
  - !ruby/object:Gem::Dependency
16
16
  name: activesupport
@@ -19,6 +19,9 @@ dependencies:
19
19
  - - ">="
20
20
  - !ruby/object:Gem::Version
21
21
  version: '4.2'
22
+ - - "<"
23
+ - !ruby/object:Gem::Version
24
+ version: '5.1'
22
25
  type: :runtime
23
26
  prerelease: false
24
27
  version_requirements: !ruby/object:Gem::Requirement
@@ -26,20 +29,37 @@ dependencies:
26
29
  - - ">="
27
30
  - !ruby/object:Gem::Version
28
31
  version: '4.2'
32
+ - - "<"
33
+ - !ruby/object:Gem::Version
34
+ version: '5.1'
35
+ - !ruby/object:Gem::Dependency
36
+ name: colorize
37
+ requirement: !ruby/object:Gem::Requirement
38
+ requirements:
39
+ - - ">="
40
+ - !ruby/object:Gem::Version
41
+ version: '0.8'
42
+ type: :runtime
43
+ prerelease: false
44
+ version_requirements: !ruby/object:Gem::Requirement
45
+ requirements:
46
+ - - ">="
47
+ - !ruby/object:Gem::Version
48
+ version: '0.8'
29
49
  - !ruby/object:Gem::Dependency
30
- name: commander
50
+ name: parallel
31
51
  requirement: !ruby/object:Gem::Requirement
32
52
  requirements:
33
53
  - - "~>"
34
54
  - !ruby/object:Gem::Version
35
- version: '4.4'
55
+ version: '1.9'
36
56
  type: :runtime
37
57
  prerelease: false
38
58
  version_requirements: !ruby/object:Gem::Requirement
39
59
  requirements:
40
60
  - - "~>"
41
61
  - !ruby/object:Gem::Version
42
- version: '4.4'
62
+ version: '1.9'
43
63
  - !ruby/object:Gem::Dependency
44
64
  name: sequel
45
65
  requirement: !ruby/object:Gem::Requirement
@@ -54,6 +74,20 @@ dependencies:
54
74
  - - "~>"
55
75
  - !ruby/object:Gem::Version
56
76
  version: '4.33'
77
+ - !ruby/object:Gem::Dependency
78
+ name: redis
79
+ requirement: !ruby/object:Gem::Requirement
80
+ requirements:
81
+ - - "~>"
82
+ - !ruby/object:Gem::Version
83
+ version: '3.3'
84
+ type: :runtime
85
+ prerelease: false
86
+ version_requirements: !ruby/object:Gem::Requirement
87
+ requirements:
88
+ - - "~>"
89
+ - !ruby/object:Gem::Version
90
+ version: '3.3'
57
91
  description:
58
92
  email: r.sprotte@ub.uni-paderborn.de
59
93
  executables:
@@ -82,6 +116,10 @@ files:
82
116
  - lib/metacrunch/job/buffer.rb
83
117
  - lib/metacrunch/job/dsl.rb
84
118
  - lib/metacrunch/job/dsl/option_support.rb
119
+ - lib/metacrunch/parallel_processable_reader.rb
120
+ - lib/metacrunch/redis.rb
121
+ - lib/metacrunch/redis/queue_reader.rb
122
+ - lib/metacrunch/redis/queue_writer.rb
85
123
  - lib/metacrunch/test_utils.rb
86
124
  - lib/metacrunch/test_utils/dummy_callable.rb
87
125
  - lib/metacrunch/test_utils/dummy_destination.rb