metacrunch 3.0.3 → 3.1.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.travis.yml +1 -1
- data/Readme.md +41 -6
- data/lib/metacrunch/cli.rb +86 -36
- data/lib/metacrunch/db/reader.rb +21 -3
- data/lib/metacrunch/db/writer.rb +7 -3
- data/lib/metacrunch/job/dsl/option_support.rb +1 -1
- data/lib/metacrunch/job.rb +17 -3
- data/lib/metacrunch/parallel_processable_reader.rb +21 -0
- data/lib/metacrunch/redis/queue_reader.rb +37 -0
- data/lib/metacrunch/redis/queue_writer.rb +26 -0
- data/lib/metacrunch/redis.rb +8 -0
- data/lib/metacrunch/version.rb +1 -1
- data/lib/metacrunch.rb +4 -2
- data/metacrunch.gemspec +4 -2
- metadata +43 -5
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: d90bf4e6defeb611ac43d6580734530b536b720c
|
4
|
+
data.tar.gz: bbabcd407de1dbad2a36f6a9fec9aa1a9f3a1fc5
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 330a3a2d2dd05198b60b4595c712a90baf64c725116492926c163903a276f9389ebaa4803e96489392a8a8b659a2cb8c1982d626dc2a0555a98e8c4887d9b96e
|
7
|
+
data.tar.gz: 151f47915a0bf4527f6b027ca241f3046d9ea01ef94d70fabcc4c8aa3b17dfd1b79871fbabff0e7db6ab58ca9737ec7aeb3e3075a28df6b784f90199db59f21a
|
data/.travis.yml
CHANGED
data/Readme.md
CHANGED
@@ -8,6 +8,7 @@ metacrunch
|
|
8
8
|
metacrunch is a simple and lightweight data processing and ETL ([Extract-Transform-Load](http://en.wikipedia.org/wiki/Extract,_transform,_load))
|
9
9
|
toolkit for Ruby.
|
10
10
|
|
11
|
+
**NOTE: THIS README IS FOR THE MASTER BRANCH. CHECK THE [RELEASES-PAGE](https://github.com/ubpb/metacrunch/releases) TO SEE THE README FOR THE RELEVANT RELEASES**
|
11
12
|
|
12
13
|
Installation
|
13
14
|
------------
|
@@ -22,13 +23,14 @@ Creating ETL jobs
|
|
22
23
|
|
23
24
|
The basic idea behind an ETL job in metacrunch is the concept of a data processing pipeline. Each ETL job reads data from one or more **sources** (extract step), runs one or more **transformations** (transform step) on the data and finally writes the transformed data to one or more **destinations** (load step).
|
24
25
|
|
25
|
-
metacrunch provides you with a simple DSL to define and run such ETL jobs. Just create a text file with the extension `.metacrunch
|
26
|
+
metacrunch provides you with a simple DSL to define and run such ETL jobs in Ruby. Just create a text file with the extension `.metacrunch` and [run it](#running-etl-jobs) with the provided `metacrunch` CLI command. *Note: The extension doesn't really matter but you should avoid `.rb` to not loading them by mistake from another Ruby component.*
|
26
27
|
|
27
28
|
Let's walk through the main steps of creating ETL jobs with metacrunch. For a collection of working examples check out our [metacrunch-demo](https://github.com/ubpb/metacrunch-demo) repo.
|
28
29
|
|
29
30
|
#### It's Ruby
|
30
31
|
|
31
|
-
Every `.metacrunch` job
|
32
|
+
Every `.metacrunch` job is a regular Ruby file and you can use any valid Ruby code like declaring methods, classes, variables, requiring other Ruby
|
33
|
+
files and so on.
|
32
34
|
|
33
35
|
```ruby
|
34
36
|
# File: my_etl_job.metacrunch
|
@@ -130,7 +132,40 @@ post_process MyCallable.new
|
|
130
132
|
|
131
133
|
#### Defining options
|
132
134
|
|
133
|
-
|
135
|
+
metacrunch has build-in support to parameterize your jobs. Using the `option` helper, you can declare options that can be set/overridden by the CLI when [running your jobs](#running-etl-jobs).
|
136
|
+
|
137
|
+
```ruby
|
138
|
+
options do
|
139
|
+
add :number_of_processes, "-n", "--no-of-processes N", "Number of processes", default: 2
|
140
|
+
add :database_url, "-d", "--database URL", "Database connection URL", required: true
|
141
|
+
end
|
142
|
+
```
|
143
|
+
|
144
|
+
In this example we declare two options `number_of_processes` and `database_url`. `number_of_processes` defaults to 2, whereas `database_url` has no default and is required. In your job file you can access the option values using the `options` Hash. E.g. `options[:number_of_processes]`.
|
145
|
+
|
146
|
+
To set/override these options use the command line.
|
147
|
+
|
148
|
+
```
|
149
|
+
$ bundle exec metacrunch my_etl_job.metacrunch @@ --no-of-processes 4
|
150
|
+
```
|
151
|
+
|
152
|
+
This will set the `options[:number_of_processes]` to `4`.
|
153
|
+
|
154
|
+
To get a list of available options for a job, use `--help` on the command line.
|
155
|
+
|
156
|
+
```
|
157
|
+
$ bundle exec metacrunch my_etl_job.metacrunch @@ --help
|
158
|
+
|
159
|
+
Usage: metacrunch run [options] JOB_FILE @@ [job-options] [ARGS]
|
160
|
+
Job options:
|
161
|
+
-n, --no-of-processes N Number of processes
|
162
|
+
DEFAULT: 2
|
163
|
+
-d, --database URL Database connection URL
|
164
|
+
REQUIRED
|
165
|
+
```
|
166
|
+
|
167
|
+
To learn more about defining options take a look at the [reference below](#defining-job-options).
|
168
|
+
|
134
169
|
|
135
170
|
Running ETL jobs
|
136
171
|
----------------
|
@@ -139,7 +174,7 @@ metacrunch comes with a handy command line tool. In a terminal use
|
|
139
174
|
|
140
175
|
|
141
176
|
```
|
142
|
-
$ metacrunch
|
177
|
+
$ metacrunch my_etl_job.metacrunch
|
143
178
|
```
|
144
179
|
|
145
180
|
to run a job.
|
@@ -147,7 +182,7 @@ to run a job.
|
|
147
182
|
If you use [Bundler](http://bundler.io) to manage dependencies for your jobs make sure to change into the directory where your Gemfile is (or set BUNDLE_GEMFILE environment variable) and run metacrunch with `bundle exec`.
|
148
183
|
|
149
184
|
```
|
150
|
-
$ bundle exec metacrunch
|
185
|
+
$ bundle exec metacrunch my_etl_job.metacrunch
|
151
186
|
```
|
152
187
|
|
153
188
|
Depending on your environment `bundle exec` may not be required (e.g. you have rubygems-bundler installed) but we recommend using it whenever you have a Gemfile you like to use. When using Bundler make sure to add `gem "metacrunch"` to the Gemfile.
|
@@ -157,7 +192,7 @@ To pass options to the job, separate job options from the metacrunch command opt
|
|
157
192
|
Use the following syntax
|
158
193
|
|
159
194
|
```
|
160
|
-
$ [bundle exec] metacrunch
|
195
|
+
$ [bundle exec] metacrunch [COMMAND_OPTIONS] JOB_FILE [@@ [JOB_OPTIONS] [JOB_ARGS...]]
|
161
196
|
```
|
162
197
|
|
163
198
|
|
data/lib/metacrunch/cli.rb
CHANGED
@@ -1,60 +1,110 @@
|
|
1
|
+
require "optparse"
|
2
|
+
|
1
3
|
module Metacrunch
|
2
4
|
class Cli
|
3
5
|
ARGS_SEPERATOR = "@@"
|
4
6
|
|
5
7
|
def run
|
6
|
-
|
7
|
-
|
8
|
-
|
8
|
+
job_files = global_parser.parse!(global_argv)
|
9
|
+
|
10
|
+
run!(job_files)
|
9
11
|
end
|
10
12
|
|
11
13
|
private
|
12
|
-
def
|
13
|
-
@
|
14
|
-
|
14
|
+
def global_parser
|
15
|
+
@global_parser ||= OptionParser.new do |opts|
|
16
|
+
opts.banner = <<-BANNER.strip_heredoc
|
17
|
+
#{ColorizedString["Usage:"].bold}
|
18
|
+
|
19
|
+
metacrunch [options] JOB_FILE @@ [job-options] [ARGS...]
|
20
|
+
|
21
|
+
#{ColorizedString["Options:"].bold}
|
22
|
+
BANNER
|
23
|
+
|
24
|
+
opts.on("-v", "--version", "Show metacrunch version and exit") do
|
25
|
+
show_version
|
26
|
+
end
|
15
27
|
|
16
|
-
|
17
|
-
|
18
|
-
|
19
|
-
|
20
|
-
|
28
|
+
opts.on("-n INTEGER", "--number-of-processes INTEGER", Integer, "Number of parallel processes to run the job. Source needs to support this. DEFAULT: 1") do |n|
|
29
|
+
error("--number-of-procs must be > 0") if n <= 0
|
30
|
+
global_options[:number_of_processes] = n
|
31
|
+
end
|
32
|
+
|
33
|
+
opts.separator "\n"
|
34
|
+
end
|
21
35
|
end
|
22
36
|
|
23
|
-
def
|
24
|
-
|
37
|
+
def global_options
|
38
|
+
@global_options ||= {
|
39
|
+
number_of_processes: 1
|
40
|
+
}
|
25
41
|
end
|
26
42
|
|
27
|
-
def
|
28
|
-
|
29
|
-
|
30
|
-
|
43
|
+
def show_version
|
44
|
+
puts Metacrunch::VERSION
|
45
|
+
exit(0)
|
46
|
+
end
|
31
47
|
|
32
|
-
|
33
|
-
|
34
|
-
|
35
|
-
|
36
|
-
|
37
|
-
say "You must provide exactly one job description file."
|
38
|
-
else
|
39
|
-
filename = File.expand_path(filenames.first)
|
40
|
-
dir = File.dirname(filename)
|
48
|
+
def error(message)
|
49
|
+
puts ColorizedString["Error: #{message}\n"].red.bold
|
50
|
+
puts global_parser.help
|
51
|
+
exit(0)
|
52
|
+
end
|
41
53
|
|
42
|
-
|
43
|
-
|
44
|
-
|
45
|
-
|
46
|
-
|
54
|
+
def global_argv
|
55
|
+
index = ARGV.index(ARGS_SEPERATOR)
|
56
|
+
if index == 0
|
57
|
+
[]
|
58
|
+
else
|
59
|
+
@global_argv ||= index ? ARGV[0..index-1] : ARGV
|
47
60
|
end
|
48
61
|
end
|
49
62
|
|
50
|
-
def
|
63
|
+
def job_argv
|
51
64
|
index = ARGV.index(ARGS_SEPERATOR)
|
52
|
-
@
|
65
|
+
@job_argv ||= index ? ARGV[index+1..-1] : nil
|
53
66
|
end
|
54
67
|
|
55
|
-
def
|
56
|
-
|
57
|
-
|
68
|
+
def run!(job_files)
|
69
|
+
if job_files.first == "run"
|
70
|
+
puts ColorizedString["WARN: Using 'run' is deprecated. Just use 'metacrunch [options] JOB_FILE @@ [job-options] [ARGS...]'\n"].yellow.bold
|
71
|
+
job_files = job_files[1..-1]
|
72
|
+
end
|
73
|
+
|
74
|
+
if job_files.empty?
|
75
|
+
error "You need to provide a job file."
|
76
|
+
elsif job_files.count > 1
|
77
|
+
error "You must provide exactly one job file."
|
78
|
+
else
|
79
|
+
job_filename = File.expand_path(job_files.first)
|
80
|
+
dir = File.dirname(job_filename)
|
81
|
+
|
82
|
+
Dir.chdir(dir) do
|
83
|
+
run_job!(job_filename)
|
84
|
+
end
|
85
|
+
end
|
86
|
+
end
|
87
|
+
|
88
|
+
def run_job!(job_filename)
|
89
|
+
if global_options[:number_of_processes] > 1
|
90
|
+
process_indicies = (0..(global_options[:number_of_processes] - 1)).to_a
|
91
|
+
|
92
|
+
Parallel.each(process_indicies) do |process_index|
|
93
|
+
Metacrunch::Job.define(
|
94
|
+
File.read(job_filename),
|
95
|
+
filename: job_filename,
|
96
|
+
args: job_argv,
|
97
|
+
number_of_processes: global_options[:number_of_processes],
|
98
|
+
process_index: process_index
|
99
|
+
).run
|
100
|
+
end
|
101
|
+
else
|
102
|
+
Metacrunch::Job.define(
|
103
|
+
File.read(job_filename),
|
104
|
+
filename: job_filename,
|
105
|
+
args: job_argv
|
106
|
+
).run
|
107
|
+
end
|
58
108
|
end
|
59
109
|
|
60
110
|
end
|
data/lib/metacrunch/db/reader.rb
CHANGED
@@ -1,5 +1,8 @@
|
|
1
|
+
require "metacrunch/db"
|
2
|
+
|
1
3
|
module Metacrunch
|
2
4
|
class Db::Reader
|
5
|
+
include Metacrunch::ParallelProcessableReader
|
3
6
|
|
4
7
|
def initialize(database_connection_or_url, dataset_proc, options = {})
|
5
8
|
@rows_per_fetch = options.delete(:rows_per_fetch) || 1000
|
@@ -10,14 +13,29 @@ module Metacrunch
|
|
10
13
|
database_connection_or_url
|
11
14
|
end
|
12
15
|
|
13
|
-
@dataset = dataset_proc.call(@db)
|
16
|
+
@dataset = dataset_proc.call(@db).unlimited
|
17
|
+
@total_numbers_of_records = @dataset.count
|
18
|
+
|
19
|
+
unless @dataset.opts[:order]
|
20
|
+
raise ArgumentError, "Metacrunch::Db::Reader requires the dataset be ordered."
|
21
|
+
end
|
14
22
|
end
|
15
23
|
|
16
24
|
def each(&block)
|
17
25
|
return enum_for(__method__) unless block_given?
|
18
26
|
|
19
|
-
@
|
20
|
-
|
27
|
+
@db.transaction do
|
28
|
+
offset = (-number_of_processes * @rows_per_fetch) + (process_index * @rows_per_fetch)
|
29
|
+
|
30
|
+
loop do
|
31
|
+
offset = offset + (number_of_processes * @rows_per_fetch)
|
32
|
+
|
33
|
+
@dataset.limit(@rows_per_fetch).offset(offset).each do |row|
|
34
|
+
yield(row)
|
35
|
+
end
|
36
|
+
|
37
|
+
break if offset + @rows_per_fetch >= @total_numbers_of_records
|
38
|
+
end
|
21
39
|
end
|
22
40
|
|
23
41
|
self
|
data/lib/metacrunch/db/writer.rb
CHANGED
@@ -1,9 +1,13 @@
|
|
1
|
+
require "metacrunch/db"
|
2
|
+
|
1
3
|
module Metacrunch
|
2
4
|
class Db::Writer
|
3
5
|
|
4
6
|
def initialize(database_connection_or_url, dataset_proc, options = {})
|
5
|
-
@use_upsert
|
6
|
-
@id_key
|
7
|
+
@use_upsert = options.delete(:use_upsert) || false
|
8
|
+
@id_key = options.delete(:id_key) || :id
|
9
|
+
@isolation_level = options.delete(:isolation_level) || :repeatable
|
10
|
+
@transaction_retries = options.delete(:transaction_retries) || 5
|
7
11
|
|
8
12
|
@db = if database_connection_or_url.is_a?(String)
|
9
13
|
Sequel.connect(database_connection_or_url, options)
|
@@ -16,7 +20,7 @@ module Metacrunch
|
|
16
20
|
|
17
21
|
def write(data)
|
18
22
|
if data.is_a?(Array)
|
19
|
-
@db.transaction do
|
23
|
+
@db.transaction(isolation: @isolation_level, num_retries: @transaction_retries) do
|
20
24
|
data.each{|d| insert_or_upsert(d) }
|
21
25
|
end
|
22
26
|
else
|
@@ -36,7 +36,7 @@ module Metacrunch
|
|
36
36
|
|
37
37
|
def parser
|
38
38
|
@parser ||= OptionParser.new do |parser|
|
39
|
-
parser.banner = "Usage: metacrunch
|
39
|
+
parser.banner = "Usage: metacrunch [options] JOB_FILE @@ [job-options] [ARGS]\nJob options:"
|
40
40
|
end
|
41
41
|
end
|
42
42
|
|
data/lib/metacrunch/job.rb
CHANGED
@@ -6,14 +6,16 @@ module Metacrunch
|
|
6
6
|
attr_reader :builder, :args
|
7
7
|
|
8
8
|
class << self
|
9
|
-
def define(file_content = nil, filename: nil, args: nil, &block)
|
10
|
-
self.new(file_content, filename: filename, args: args, &block)
|
9
|
+
def define(file_content = nil, filename: nil, args: nil, number_of_processes: 1, process_index: 0, &block)
|
10
|
+
self.new(file_content, filename: filename, args: args, number_of_processes: number_of_processes, process_index: process_index, &block)
|
11
11
|
end
|
12
12
|
end
|
13
13
|
|
14
|
-
def initialize(file_content = nil, filename: nil, args: nil, &block)
|
14
|
+
def initialize(file_content = nil, filename: nil, args: nil, number_of_processes: 1, process_index: 0, &block)
|
15
15
|
@builder = Dsl.new(self)
|
16
16
|
@args = args
|
17
|
+
@number_of_processes = number_of_processes
|
18
|
+
@process_index = process_index
|
17
19
|
|
18
20
|
if file_content
|
19
21
|
@builder.instance_eval(file_content, filename || "")
|
@@ -109,6 +111,18 @@ module Metacrunch
|
|
109
111
|
|
110
112
|
def run_transformations
|
111
113
|
sources.each do |source|
|
114
|
+
# Setup parallel processing
|
115
|
+
if @number_of_processes > 1
|
116
|
+
if source.class.included_modules.include?(Metacrunch::ParallelProcessableReader)
|
117
|
+
source.set_parallel_process_options(
|
118
|
+
number_of_processes: @number_of_processes,
|
119
|
+
process_index: @process_index
|
120
|
+
)
|
121
|
+
else
|
122
|
+
raise RuntimeError, "source does't support parallel processing"
|
123
|
+
end
|
124
|
+
end
|
125
|
+
|
112
126
|
# sources are expected to respond to `each`
|
113
127
|
source.each do |data|
|
114
128
|
run_transformations_and_write_destinations(data)
|
@@ -0,0 +1,21 @@
|
|
1
|
+
module Metacrunch
|
2
|
+
module ParallelProcessableReader
|
3
|
+
|
4
|
+
def set_parallel_process_options(number_of_processes: 1, process_index: 0)
|
5
|
+
raise ArgumentError, "number_of_processes must be >= 1" if number_of_processes < 1
|
6
|
+
raise ArgumentError, "process_index must be >= 0" if process_index < 0
|
7
|
+
|
8
|
+
@number_of_processes = number_of_processes
|
9
|
+
@process_index = process_index
|
10
|
+
end
|
11
|
+
|
12
|
+
def number_of_processes
|
13
|
+
@number_of_processes || 1
|
14
|
+
end
|
15
|
+
|
16
|
+
def process_index
|
17
|
+
@process_index || 0
|
18
|
+
end
|
19
|
+
|
20
|
+
end
|
21
|
+
end
|
@@ -0,0 +1,37 @@
|
|
1
|
+
require "metacrunch/redis"
|
2
|
+
|
3
|
+
module Metacrunch
|
4
|
+
class Redis::QueueReader
|
5
|
+
|
6
|
+
def initialize(redis_connection_or_url, queue_name, options = {})
|
7
|
+
@queue_name = queue_name
|
8
|
+
raise ArgumentError, "queue_name must be a string" unless queue_name.is_a?(String)
|
9
|
+
|
10
|
+
@blocking_mode = options.delete(:blocking) || false
|
11
|
+
|
12
|
+
@redis = if redis_connection_or_url.is_a?(String)
|
13
|
+
::Redis.new(url: redis_connection_or_url)
|
14
|
+
else
|
15
|
+
redis_connection_or_url
|
16
|
+
end
|
17
|
+
end
|
18
|
+
|
19
|
+
def each(&block)
|
20
|
+
return enum_for(__method__) unless block_given?
|
21
|
+
|
22
|
+
if @blocking_mode
|
23
|
+
while true
|
24
|
+
result = @redis.blpop(@queue_name)
|
25
|
+
yield JSON.parse(result[1]) if result
|
26
|
+
end
|
27
|
+
else
|
28
|
+
while result = @redis.lpop(@queue_name)
|
29
|
+
yield JSON.parse(result)
|
30
|
+
end
|
31
|
+
end
|
32
|
+
|
33
|
+
self
|
34
|
+
end
|
35
|
+
|
36
|
+
end
|
37
|
+
end
|
@@ -0,0 +1,26 @@
|
|
1
|
+
require "metacrunch/redis"
|
2
|
+
|
3
|
+
module Metacrunch
|
4
|
+
class Redis::QueueWriter
|
5
|
+
|
6
|
+
def initialize(redis_connection_or_url, queue_name, options = {})
|
7
|
+
@queue_name = queue_name
|
8
|
+
raise ArgumentError, "queue_name must be a string" unless queue_name.is_a?(String)
|
9
|
+
|
10
|
+
@redis = if redis_connection_or_url.is_a?(String)
|
11
|
+
::Redis.new(url: redis_connection_or_url)
|
12
|
+
else
|
13
|
+
redis_connection_or_url
|
14
|
+
end
|
15
|
+
end
|
16
|
+
|
17
|
+
def write(data)
|
18
|
+
@redis.rpush(@queue_name, data.to_json)
|
19
|
+
end
|
20
|
+
|
21
|
+
def close
|
22
|
+
@redis.close if @redis
|
23
|
+
end
|
24
|
+
|
25
|
+
end
|
26
|
+
end
|
data/lib/metacrunch/version.rb
CHANGED
data/lib/metacrunch.rb
CHANGED
@@ -1,12 +1,14 @@
|
|
1
1
|
require "active_support"
|
2
2
|
require "active_support/core_ext"
|
3
|
-
require "
|
4
|
-
require "
|
3
|
+
require "colorized_string"
|
4
|
+
require "parallel"
|
5
5
|
|
6
6
|
module Metacrunch
|
7
7
|
require_relative "metacrunch/version"
|
8
8
|
require_relative "metacrunch/cli"
|
9
9
|
require_relative "metacrunch/job"
|
10
|
+
require_relative "metacrunch/parallel_processable_reader"
|
10
11
|
require_relative "metacrunch/fs"
|
11
12
|
require_relative "metacrunch/db"
|
13
|
+
require_relative "metacrunch/redis"
|
12
14
|
end
|
data/metacrunch.gemspec
CHANGED
@@ -17,7 +17,9 @@ Gem::Specification.new do |spec|
|
|
17
17
|
spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
|
18
18
|
spec.require_paths = ["lib"]
|
19
19
|
|
20
|
-
spec.add_dependency "activesupport", ">= 4.2"
|
21
|
-
spec.add_dependency "
|
20
|
+
spec.add_dependency "activesupport", ">= 4.2", "< 5.1"
|
21
|
+
spec.add_dependency "colorize", ">= 0.8"
|
22
|
+
spec.add_dependency "parallel", "~> 1.9"
|
22
23
|
spec.add_dependency "sequel", "~> 4.33"
|
24
|
+
spec.add_dependency "redis", "~> 3.3"
|
23
25
|
end
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: metacrunch
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 3.0
|
4
|
+
version: 3.1.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- René Sprotte
|
@@ -10,7 +10,7 @@ authors:
|
|
10
10
|
autorequire:
|
11
11
|
bindir: exe
|
12
12
|
cert_chain: []
|
13
|
-
date: 2016-07-
|
13
|
+
date: 2016-07-21 00:00:00.000000000 Z
|
14
14
|
dependencies:
|
15
15
|
- !ruby/object:Gem::Dependency
|
16
16
|
name: activesupport
|
@@ -19,6 +19,9 @@ dependencies:
|
|
19
19
|
- - ">="
|
20
20
|
- !ruby/object:Gem::Version
|
21
21
|
version: '4.2'
|
22
|
+
- - "<"
|
23
|
+
- !ruby/object:Gem::Version
|
24
|
+
version: '5.1'
|
22
25
|
type: :runtime
|
23
26
|
prerelease: false
|
24
27
|
version_requirements: !ruby/object:Gem::Requirement
|
@@ -26,20 +29,37 @@ dependencies:
|
|
26
29
|
- - ">="
|
27
30
|
- !ruby/object:Gem::Version
|
28
31
|
version: '4.2'
|
32
|
+
- - "<"
|
33
|
+
- !ruby/object:Gem::Version
|
34
|
+
version: '5.1'
|
35
|
+
- !ruby/object:Gem::Dependency
|
36
|
+
name: colorize
|
37
|
+
requirement: !ruby/object:Gem::Requirement
|
38
|
+
requirements:
|
39
|
+
- - ">="
|
40
|
+
- !ruby/object:Gem::Version
|
41
|
+
version: '0.8'
|
42
|
+
type: :runtime
|
43
|
+
prerelease: false
|
44
|
+
version_requirements: !ruby/object:Gem::Requirement
|
45
|
+
requirements:
|
46
|
+
- - ">="
|
47
|
+
- !ruby/object:Gem::Version
|
48
|
+
version: '0.8'
|
29
49
|
- !ruby/object:Gem::Dependency
|
30
|
-
name:
|
50
|
+
name: parallel
|
31
51
|
requirement: !ruby/object:Gem::Requirement
|
32
52
|
requirements:
|
33
53
|
- - "~>"
|
34
54
|
- !ruby/object:Gem::Version
|
35
|
-
version: '
|
55
|
+
version: '1.9'
|
36
56
|
type: :runtime
|
37
57
|
prerelease: false
|
38
58
|
version_requirements: !ruby/object:Gem::Requirement
|
39
59
|
requirements:
|
40
60
|
- - "~>"
|
41
61
|
- !ruby/object:Gem::Version
|
42
|
-
version: '
|
62
|
+
version: '1.9'
|
43
63
|
- !ruby/object:Gem::Dependency
|
44
64
|
name: sequel
|
45
65
|
requirement: !ruby/object:Gem::Requirement
|
@@ -54,6 +74,20 @@ dependencies:
|
|
54
74
|
- - "~>"
|
55
75
|
- !ruby/object:Gem::Version
|
56
76
|
version: '4.33'
|
77
|
+
- !ruby/object:Gem::Dependency
|
78
|
+
name: redis
|
79
|
+
requirement: !ruby/object:Gem::Requirement
|
80
|
+
requirements:
|
81
|
+
- - "~>"
|
82
|
+
- !ruby/object:Gem::Version
|
83
|
+
version: '3.3'
|
84
|
+
type: :runtime
|
85
|
+
prerelease: false
|
86
|
+
version_requirements: !ruby/object:Gem::Requirement
|
87
|
+
requirements:
|
88
|
+
- - "~>"
|
89
|
+
- !ruby/object:Gem::Version
|
90
|
+
version: '3.3'
|
57
91
|
description:
|
58
92
|
email: r.sprotte@ub.uni-paderborn.de
|
59
93
|
executables:
|
@@ -82,6 +116,10 @@ files:
|
|
82
116
|
- lib/metacrunch/job/buffer.rb
|
83
117
|
- lib/metacrunch/job/dsl.rb
|
84
118
|
- lib/metacrunch/job/dsl/option_support.rb
|
119
|
+
- lib/metacrunch/parallel_processable_reader.rb
|
120
|
+
- lib/metacrunch/redis.rb
|
121
|
+
- lib/metacrunch/redis/queue_reader.rb
|
122
|
+
- lib/metacrunch/redis/queue_writer.rb
|
85
123
|
- lib/metacrunch/test_utils.rb
|
86
124
|
- lib/metacrunch/test_utils/dummy_callable.rb
|
87
125
|
- lib/metacrunch/test_utils/dummy_destination.rb
|