metacrunch 3.0.3 → 3.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.travis.yml +1 -1
- data/Readme.md +41 -6
- data/lib/metacrunch/cli.rb +86 -36
- data/lib/metacrunch/db/reader.rb +21 -3
- data/lib/metacrunch/db/writer.rb +7 -3
- data/lib/metacrunch/job/dsl/option_support.rb +1 -1
- data/lib/metacrunch/job.rb +17 -3
- data/lib/metacrunch/parallel_processable_reader.rb +21 -0
- data/lib/metacrunch/redis/queue_reader.rb +37 -0
- data/lib/metacrunch/redis/queue_writer.rb +26 -0
- data/lib/metacrunch/redis.rb +8 -0
- data/lib/metacrunch/version.rb +1 -1
- data/lib/metacrunch.rb +4 -2
- data/metacrunch.gemspec +4 -2
- metadata +43 -5
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: d90bf4e6defeb611ac43d6580734530b536b720c
|
4
|
+
data.tar.gz: bbabcd407de1dbad2a36f6a9fec9aa1a9f3a1fc5
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 330a3a2d2dd05198b60b4595c712a90baf64c725116492926c163903a276f9389ebaa4803e96489392a8a8b659a2cb8c1982d626dc2a0555a98e8c4887d9b96e
|
7
|
+
data.tar.gz: 151f47915a0bf4527f6b027ca241f3046d9ea01ef94d70fabcc4c8aa3b17dfd1b79871fbabff0e7db6ab58ca9737ec7aeb3e3075a28df6b784f90199db59f21a
|
data/.travis.yml
CHANGED
data/Readme.md
CHANGED
@@ -8,6 +8,7 @@ metacrunch
|
|
8
8
|
metacrunch is a simple and lightweight data processing and ETL ([Extract-Transform-Load](http://en.wikipedia.org/wiki/Extract,_transform,_load))
|
9
9
|
toolkit for Ruby.
|
10
10
|
|
11
|
+
**NOTE: THIS README IS FOR THE MASTER BRANCH. CHECK THE [RELEASES-PAGE](https://github.com/ubpb/metacrunch/releases) TO SEE THE README FOR THE RELEVANT RELEASES**
|
11
12
|
|
12
13
|
Installation
|
13
14
|
------------
|
@@ -22,13 +23,14 @@ Creating ETL jobs
|
|
22
23
|
|
23
24
|
The basic idea behind an ETL job in metacrunch is the concept of a data processing pipeline. Each ETL job reads data from one or more **sources** (extract step), runs one or more **transformations** (transform step) on the data and finally writes the transformed data to one or more **destinations** (load step).
|
24
25
|
|
25
|
-
metacrunch provides you with a simple DSL to define and run such ETL jobs. Just create a text file with the extension `.metacrunch
|
26
|
+
metacrunch provides you with a simple DSL to define and run such ETL jobs in Ruby. Just create a text file with the extension `.metacrunch` and [run it](#running-etl-jobs) with the provided `metacrunch` CLI command. *Note: The extension doesn't really matter but you should avoid `.rb` to not loading them by mistake from another Ruby component.*
|
26
27
|
|
27
28
|
Let's walk through the main steps of creating ETL jobs with metacrunch. For a collection of working examples check out our [metacrunch-demo](https://github.com/ubpb/metacrunch-demo) repo.
|
28
29
|
|
29
30
|
#### It's Ruby
|
30
31
|
|
31
|
-
Every `.metacrunch` job
|
32
|
+
Every `.metacrunch` job is a regular Ruby file and you can use any valid Ruby code like declaring methods, classes, variables, requiring other Ruby
|
33
|
+
files and so on.
|
32
34
|
|
33
35
|
```ruby
|
34
36
|
# File: my_etl_job.metacrunch
|
@@ -130,7 +132,40 @@ post_process MyCallable.new
|
|
130
132
|
|
131
133
|
#### Defining options
|
132
134
|
|
133
|
-
|
135
|
+
metacrunch has build-in support to parameterize your jobs. Using the `option` helper, you can declare options that can be set/overridden by the CLI when [running your jobs](#running-etl-jobs).
|
136
|
+
|
137
|
+
```ruby
|
138
|
+
options do
|
139
|
+
add :number_of_processes, "-n", "--no-of-processes N", "Number of processes", default: 2
|
140
|
+
add :database_url, "-d", "--database URL", "Database connection URL", required: true
|
141
|
+
end
|
142
|
+
```
|
143
|
+
|
144
|
+
In this example we declare two options `number_of_processes` and `database_url`. `number_of_processes` defaults to 2, whereas `database_url` has no default and is required. In your job file you can access the option values using the `options` Hash. E.g. `options[:number_of_processes]`.
|
145
|
+
|
146
|
+
To set/override these options use the command line.
|
147
|
+
|
148
|
+
```
|
149
|
+
$ bundle exec metacrunch my_etl_job.metacrunch @@ --no-of-processes 4
|
150
|
+
```
|
151
|
+
|
152
|
+
This will set the `options[:number_of_processes]` to `4`.
|
153
|
+
|
154
|
+
To get a list of available options for a job, use `--help` on the command line.
|
155
|
+
|
156
|
+
```
|
157
|
+
$ bundle exec metacrunch my_etl_job.metacrunch @@ --help
|
158
|
+
|
159
|
+
Usage: metacrunch run [options] JOB_FILE @@ [job-options] [ARGS]
|
160
|
+
Job options:
|
161
|
+
-n, --no-of-processes N Number of processes
|
162
|
+
DEFAULT: 2
|
163
|
+
-d, --database URL Database connection URL
|
164
|
+
REQUIRED
|
165
|
+
```
|
166
|
+
|
167
|
+
To learn more about defining options take a look at the [reference below](#defining-job-options).
|
168
|
+
|
134
169
|
|
135
170
|
Running ETL jobs
|
136
171
|
----------------
|
@@ -139,7 +174,7 @@ metacrunch comes with a handy command line tool. In a terminal use
|
|
139
174
|
|
140
175
|
|
141
176
|
```
|
142
|
-
$ metacrunch
|
177
|
+
$ metacrunch my_etl_job.metacrunch
|
143
178
|
```
|
144
179
|
|
145
180
|
to run a job.
|
@@ -147,7 +182,7 @@ to run a job.
|
|
147
182
|
If you use [Bundler](http://bundler.io) to manage dependencies for your jobs make sure to change into the directory where your Gemfile is (or set BUNDLE_GEMFILE environment variable) and run metacrunch with `bundle exec`.
|
148
183
|
|
149
184
|
```
|
150
|
-
$ bundle exec metacrunch
|
185
|
+
$ bundle exec metacrunch my_etl_job.metacrunch
|
151
186
|
```
|
152
187
|
|
153
188
|
Depending on your environment `bundle exec` may not be required (e.g. you have rubygems-bundler installed) but we recommend using it whenever you have a Gemfile you like to use. When using Bundler make sure to add `gem "metacrunch"` to the Gemfile.
|
@@ -157,7 +192,7 @@ To pass options to the job, separate job options from the metacrunch command opt
|
|
157
192
|
Use the following syntax
|
158
193
|
|
159
194
|
```
|
160
|
-
$ [bundle exec] metacrunch
|
195
|
+
$ [bundle exec] metacrunch [COMMAND_OPTIONS] JOB_FILE [@@ [JOB_OPTIONS] [JOB_ARGS...]]
|
161
196
|
```
|
162
197
|
|
163
198
|
|
data/lib/metacrunch/cli.rb
CHANGED
@@ -1,60 +1,110 @@
|
|
1
|
+
require "optparse"
|
2
|
+
|
1
3
|
module Metacrunch
|
2
4
|
class Cli
|
3
5
|
ARGS_SEPERATOR = "@@"
|
4
6
|
|
5
7
|
def run
|
6
|
-
|
7
|
-
|
8
|
-
|
8
|
+
job_files = global_parser.parse!(global_argv)
|
9
|
+
|
10
|
+
run!(job_files)
|
9
11
|
end
|
10
12
|
|
11
13
|
private
|
12
|
-
def
|
13
|
-
@
|
14
|
-
|
14
|
+
def global_parser
|
15
|
+
@global_parser ||= OptionParser.new do |opts|
|
16
|
+
opts.banner = <<-BANNER.strip_heredoc
|
17
|
+
#{ColorizedString["Usage:"].bold}
|
18
|
+
|
19
|
+
metacrunch [options] JOB_FILE @@ [job-options] [ARGS...]
|
20
|
+
|
21
|
+
#{ColorizedString["Options:"].bold}
|
22
|
+
BANNER
|
23
|
+
|
24
|
+
opts.on("-v", "--version", "Show metacrunch version and exit") do
|
25
|
+
show_version
|
26
|
+
end
|
15
27
|
|
16
|
-
|
17
|
-
|
18
|
-
|
19
|
-
|
20
|
-
|
28
|
+
opts.on("-n INTEGER", "--number-of-processes INTEGER", Integer, "Number of parallel processes to run the job. Source needs to support this. DEFAULT: 1") do |n|
|
29
|
+
error("--number-of-procs must be > 0") if n <= 0
|
30
|
+
global_options[:number_of_processes] = n
|
31
|
+
end
|
32
|
+
|
33
|
+
opts.separator "\n"
|
34
|
+
end
|
21
35
|
end
|
22
36
|
|
23
|
-
def
|
24
|
-
|
37
|
+
def global_options
|
38
|
+
@global_options ||= {
|
39
|
+
number_of_processes: 1
|
40
|
+
}
|
25
41
|
end
|
26
42
|
|
27
|
-
def
|
28
|
-
|
29
|
-
|
30
|
-
|
43
|
+
def show_version
|
44
|
+
puts Metacrunch::VERSION
|
45
|
+
exit(0)
|
46
|
+
end
|
31
47
|
|
32
|
-
|
33
|
-
|
34
|
-
|
35
|
-
|
36
|
-
|
37
|
-
say "You must provide exactly one job description file."
|
38
|
-
else
|
39
|
-
filename = File.expand_path(filenames.first)
|
40
|
-
dir = File.dirname(filename)
|
48
|
+
def error(message)
|
49
|
+
puts ColorizedString["Error: #{message}\n"].red.bold
|
50
|
+
puts global_parser.help
|
51
|
+
exit(0)
|
52
|
+
end
|
41
53
|
|
42
|
-
|
43
|
-
|
44
|
-
|
45
|
-
|
46
|
-
|
54
|
+
def global_argv
|
55
|
+
index = ARGV.index(ARGS_SEPERATOR)
|
56
|
+
if index == 0
|
57
|
+
[]
|
58
|
+
else
|
59
|
+
@global_argv ||= index ? ARGV[0..index-1] : ARGV
|
47
60
|
end
|
48
61
|
end
|
49
62
|
|
50
|
-
def
|
63
|
+
def job_argv
|
51
64
|
index = ARGV.index(ARGS_SEPERATOR)
|
52
|
-
@
|
65
|
+
@job_argv ||= index ? ARGV[index+1..-1] : nil
|
53
66
|
end
|
54
67
|
|
55
|
-
def
|
56
|
-
|
57
|
-
|
68
|
+
def run!(job_files)
|
69
|
+
if job_files.first == "run"
|
70
|
+
puts ColorizedString["WARN: Using 'run' is deprecated. Just use 'metacrunch [options] JOB_FILE @@ [job-options] [ARGS...]'\n"].yellow.bold
|
71
|
+
job_files = job_files[1..-1]
|
72
|
+
end
|
73
|
+
|
74
|
+
if job_files.empty?
|
75
|
+
error "You need to provide a job file."
|
76
|
+
elsif job_files.count > 1
|
77
|
+
error "You must provide exactly one job file."
|
78
|
+
else
|
79
|
+
job_filename = File.expand_path(job_files.first)
|
80
|
+
dir = File.dirname(job_filename)
|
81
|
+
|
82
|
+
Dir.chdir(dir) do
|
83
|
+
run_job!(job_filename)
|
84
|
+
end
|
85
|
+
end
|
86
|
+
end
|
87
|
+
|
88
|
+
def run_job!(job_filename)
|
89
|
+
if global_options[:number_of_processes] > 1
|
90
|
+
process_indicies = (0..(global_options[:number_of_processes] - 1)).to_a
|
91
|
+
|
92
|
+
Parallel.each(process_indicies) do |process_index|
|
93
|
+
Metacrunch::Job.define(
|
94
|
+
File.read(job_filename),
|
95
|
+
filename: job_filename,
|
96
|
+
args: job_argv,
|
97
|
+
number_of_processes: global_options[:number_of_processes],
|
98
|
+
process_index: process_index
|
99
|
+
).run
|
100
|
+
end
|
101
|
+
else
|
102
|
+
Metacrunch::Job.define(
|
103
|
+
File.read(job_filename),
|
104
|
+
filename: job_filename,
|
105
|
+
args: job_argv
|
106
|
+
).run
|
107
|
+
end
|
58
108
|
end
|
59
109
|
|
60
110
|
end
|
data/lib/metacrunch/db/reader.rb
CHANGED
@@ -1,5 +1,8 @@
|
|
1
|
+
require "metacrunch/db"
|
2
|
+
|
1
3
|
module Metacrunch
|
2
4
|
class Db::Reader
|
5
|
+
include Metacrunch::ParallelProcessableReader
|
3
6
|
|
4
7
|
def initialize(database_connection_or_url, dataset_proc, options = {})
|
5
8
|
@rows_per_fetch = options.delete(:rows_per_fetch) || 1000
|
@@ -10,14 +13,29 @@ module Metacrunch
|
|
10
13
|
database_connection_or_url
|
11
14
|
end
|
12
15
|
|
13
|
-
@dataset = dataset_proc.call(@db)
|
16
|
+
@dataset = dataset_proc.call(@db).unlimited
|
17
|
+
@total_numbers_of_records = @dataset.count
|
18
|
+
|
19
|
+
unless @dataset.opts[:order]
|
20
|
+
raise ArgumentError, "Metacrunch::Db::Reader requires the dataset be ordered."
|
21
|
+
end
|
14
22
|
end
|
15
23
|
|
16
24
|
def each(&block)
|
17
25
|
return enum_for(__method__) unless block_given?
|
18
26
|
|
19
|
-
@
|
20
|
-
|
27
|
+
@db.transaction do
|
28
|
+
offset = (-number_of_processes * @rows_per_fetch) + (process_index * @rows_per_fetch)
|
29
|
+
|
30
|
+
loop do
|
31
|
+
offset = offset + (number_of_processes * @rows_per_fetch)
|
32
|
+
|
33
|
+
@dataset.limit(@rows_per_fetch).offset(offset).each do |row|
|
34
|
+
yield(row)
|
35
|
+
end
|
36
|
+
|
37
|
+
break if offset + @rows_per_fetch >= @total_numbers_of_records
|
38
|
+
end
|
21
39
|
end
|
22
40
|
|
23
41
|
self
|
data/lib/metacrunch/db/writer.rb
CHANGED
@@ -1,9 +1,13 @@
|
|
1
|
+
require "metacrunch/db"
|
2
|
+
|
1
3
|
module Metacrunch
|
2
4
|
class Db::Writer
|
3
5
|
|
4
6
|
def initialize(database_connection_or_url, dataset_proc, options = {})
|
5
|
-
@use_upsert
|
6
|
-
@id_key
|
7
|
+
@use_upsert = options.delete(:use_upsert) || false
|
8
|
+
@id_key = options.delete(:id_key) || :id
|
9
|
+
@isolation_level = options.delete(:isolation_level) || :repeatable
|
10
|
+
@transaction_retries = options.delete(:transaction_retries) || 5
|
7
11
|
|
8
12
|
@db = if database_connection_or_url.is_a?(String)
|
9
13
|
Sequel.connect(database_connection_or_url, options)
|
@@ -16,7 +20,7 @@ module Metacrunch
|
|
16
20
|
|
17
21
|
def write(data)
|
18
22
|
if data.is_a?(Array)
|
19
|
-
@db.transaction do
|
23
|
+
@db.transaction(isolation: @isolation_level, num_retries: @transaction_retries) do
|
20
24
|
data.each{|d| insert_or_upsert(d) }
|
21
25
|
end
|
22
26
|
else
|
@@ -36,7 +36,7 @@ module Metacrunch
|
|
36
36
|
|
37
37
|
def parser
|
38
38
|
@parser ||= OptionParser.new do |parser|
|
39
|
-
parser.banner = "Usage: metacrunch
|
39
|
+
parser.banner = "Usage: metacrunch [options] JOB_FILE @@ [job-options] [ARGS]\nJob options:"
|
40
40
|
end
|
41
41
|
end
|
42
42
|
|
data/lib/metacrunch/job.rb
CHANGED
@@ -6,14 +6,16 @@ module Metacrunch
|
|
6
6
|
attr_reader :builder, :args
|
7
7
|
|
8
8
|
class << self
|
9
|
-
def define(file_content = nil, filename: nil, args: nil, &block)
|
10
|
-
self.new(file_content, filename: filename, args: args, &block)
|
9
|
+
def define(file_content = nil, filename: nil, args: nil, number_of_processes: 1, process_index: 0, &block)
|
10
|
+
self.new(file_content, filename: filename, args: args, number_of_processes: number_of_processes, process_index: process_index, &block)
|
11
11
|
end
|
12
12
|
end
|
13
13
|
|
14
|
-
def initialize(file_content = nil, filename: nil, args: nil, &block)
|
14
|
+
def initialize(file_content = nil, filename: nil, args: nil, number_of_processes: 1, process_index: 0, &block)
|
15
15
|
@builder = Dsl.new(self)
|
16
16
|
@args = args
|
17
|
+
@number_of_processes = number_of_processes
|
18
|
+
@process_index = process_index
|
17
19
|
|
18
20
|
if file_content
|
19
21
|
@builder.instance_eval(file_content, filename || "")
|
@@ -109,6 +111,18 @@ module Metacrunch
|
|
109
111
|
|
110
112
|
def run_transformations
|
111
113
|
sources.each do |source|
|
114
|
+
# Setup parallel processing
|
115
|
+
if @number_of_processes > 1
|
116
|
+
if source.class.included_modules.include?(Metacrunch::ParallelProcessableReader)
|
117
|
+
source.set_parallel_process_options(
|
118
|
+
number_of_processes: @number_of_processes,
|
119
|
+
process_index: @process_index
|
120
|
+
)
|
121
|
+
else
|
122
|
+
raise RuntimeError, "source does't support parallel processing"
|
123
|
+
end
|
124
|
+
end
|
125
|
+
|
112
126
|
# sources are expected to respond to `each`
|
113
127
|
source.each do |data|
|
114
128
|
run_transformations_and_write_destinations(data)
|
@@ -0,0 +1,21 @@
|
|
1
|
+
module Metacrunch
|
2
|
+
module ParallelProcessableReader
|
3
|
+
|
4
|
+
def set_parallel_process_options(number_of_processes: 1, process_index: 0)
|
5
|
+
raise ArgumentError, "number_of_processes must be >= 1" if number_of_processes < 1
|
6
|
+
raise ArgumentError, "process_index must be >= 0" if process_index < 0
|
7
|
+
|
8
|
+
@number_of_processes = number_of_processes
|
9
|
+
@process_index = process_index
|
10
|
+
end
|
11
|
+
|
12
|
+
def number_of_processes
|
13
|
+
@number_of_processes || 1
|
14
|
+
end
|
15
|
+
|
16
|
+
def process_index
|
17
|
+
@process_index || 0
|
18
|
+
end
|
19
|
+
|
20
|
+
end
|
21
|
+
end
|
@@ -0,0 +1,37 @@
|
|
1
|
+
require "metacrunch/redis"
|
2
|
+
|
3
|
+
module Metacrunch
|
4
|
+
class Redis::QueueReader
|
5
|
+
|
6
|
+
def initialize(redis_connection_or_url, queue_name, options = {})
|
7
|
+
@queue_name = queue_name
|
8
|
+
raise ArgumentError, "queue_name must be a string" unless queue_name.is_a?(String)
|
9
|
+
|
10
|
+
@blocking_mode = options.delete(:blocking) || false
|
11
|
+
|
12
|
+
@redis = if redis_connection_or_url.is_a?(String)
|
13
|
+
::Redis.new(url: redis_connection_or_url)
|
14
|
+
else
|
15
|
+
redis_connection_or_url
|
16
|
+
end
|
17
|
+
end
|
18
|
+
|
19
|
+
def each(&block)
|
20
|
+
return enum_for(__method__) unless block_given?
|
21
|
+
|
22
|
+
if @blocking_mode
|
23
|
+
while true
|
24
|
+
result = @redis.blpop(@queue_name)
|
25
|
+
yield JSON.parse(result[1]) if result
|
26
|
+
end
|
27
|
+
else
|
28
|
+
while result = @redis.lpop(@queue_name)
|
29
|
+
yield JSON.parse(result)
|
30
|
+
end
|
31
|
+
end
|
32
|
+
|
33
|
+
self
|
34
|
+
end
|
35
|
+
|
36
|
+
end
|
37
|
+
end
|
@@ -0,0 +1,26 @@
|
|
1
|
+
require "metacrunch/redis"
|
2
|
+
|
3
|
+
module Metacrunch
|
4
|
+
class Redis::QueueWriter
|
5
|
+
|
6
|
+
def initialize(redis_connection_or_url, queue_name, options = {})
|
7
|
+
@queue_name = queue_name
|
8
|
+
raise ArgumentError, "queue_name must be a string" unless queue_name.is_a?(String)
|
9
|
+
|
10
|
+
@redis = if redis_connection_or_url.is_a?(String)
|
11
|
+
::Redis.new(url: redis_connection_or_url)
|
12
|
+
else
|
13
|
+
redis_connection_or_url
|
14
|
+
end
|
15
|
+
end
|
16
|
+
|
17
|
+
def write(data)
|
18
|
+
@redis.rpush(@queue_name, data.to_json)
|
19
|
+
end
|
20
|
+
|
21
|
+
def close
|
22
|
+
@redis.close if @redis
|
23
|
+
end
|
24
|
+
|
25
|
+
end
|
26
|
+
end
|
data/lib/metacrunch/version.rb
CHANGED
data/lib/metacrunch.rb
CHANGED
@@ -1,12 +1,14 @@
|
|
1
1
|
require "active_support"
|
2
2
|
require "active_support/core_ext"
|
3
|
-
require "
|
4
|
-
require "
|
3
|
+
require "colorized_string"
|
4
|
+
require "parallel"
|
5
5
|
|
6
6
|
module Metacrunch
|
7
7
|
require_relative "metacrunch/version"
|
8
8
|
require_relative "metacrunch/cli"
|
9
9
|
require_relative "metacrunch/job"
|
10
|
+
require_relative "metacrunch/parallel_processable_reader"
|
10
11
|
require_relative "metacrunch/fs"
|
11
12
|
require_relative "metacrunch/db"
|
13
|
+
require_relative "metacrunch/redis"
|
12
14
|
end
|
data/metacrunch.gemspec
CHANGED
@@ -17,7 +17,9 @@ Gem::Specification.new do |spec|
|
|
17
17
|
spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
|
18
18
|
spec.require_paths = ["lib"]
|
19
19
|
|
20
|
-
spec.add_dependency "activesupport", ">= 4.2"
|
21
|
-
spec.add_dependency "
|
20
|
+
spec.add_dependency "activesupport", ">= 4.2", "< 5.1"
|
21
|
+
spec.add_dependency "colorize", ">= 0.8"
|
22
|
+
spec.add_dependency "parallel", "~> 1.9"
|
22
23
|
spec.add_dependency "sequel", "~> 4.33"
|
24
|
+
spec.add_dependency "redis", "~> 3.3"
|
23
25
|
end
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: metacrunch
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 3.0
|
4
|
+
version: 3.1.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- René Sprotte
|
@@ -10,7 +10,7 @@ authors:
|
|
10
10
|
autorequire:
|
11
11
|
bindir: exe
|
12
12
|
cert_chain: []
|
13
|
-
date: 2016-07-
|
13
|
+
date: 2016-07-21 00:00:00.000000000 Z
|
14
14
|
dependencies:
|
15
15
|
- !ruby/object:Gem::Dependency
|
16
16
|
name: activesupport
|
@@ -19,6 +19,9 @@ dependencies:
|
|
19
19
|
- - ">="
|
20
20
|
- !ruby/object:Gem::Version
|
21
21
|
version: '4.2'
|
22
|
+
- - "<"
|
23
|
+
- !ruby/object:Gem::Version
|
24
|
+
version: '5.1'
|
22
25
|
type: :runtime
|
23
26
|
prerelease: false
|
24
27
|
version_requirements: !ruby/object:Gem::Requirement
|
@@ -26,20 +29,37 @@ dependencies:
|
|
26
29
|
- - ">="
|
27
30
|
- !ruby/object:Gem::Version
|
28
31
|
version: '4.2'
|
32
|
+
- - "<"
|
33
|
+
- !ruby/object:Gem::Version
|
34
|
+
version: '5.1'
|
35
|
+
- !ruby/object:Gem::Dependency
|
36
|
+
name: colorize
|
37
|
+
requirement: !ruby/object:Gem::Requirement
|
38
|
+
requirements:
|
39
|
+
- - ">="
|
40
|
+
- !ruby/object:Gem::Version
|
41
|
+
version: '0.8'
|
42
|
+
type: :runtime
|
43
|
+
prerelease: false
|
44
|
+
version_requirements: !ruby/object:Gem::Requirement
|
45
|
+
requirements:
|
46
|
+
- - ">="
|
47
|
+
- !ruby/object:Gem::Version
|
48
|
+
version: '0.8'
|
29
49
|
- !ruby/object:Gem::Dependency
|
30
|
-
name:
|
50
|
+
name: parallel
|
31
51
|
requirement: !ruby/object:Gem::Requirement
|
32
52
|
requirements:
|
33
53
|
- - "~>"
|
34
54
|
- !ruby/object:Gem::Version
|
35
|
-
version: '
|
55
|
+
version: '1.9'
|
36
56
|
type: :runtime
|
37
57
|
prerelease: false
|
38
58
|
version_requirements: !ruby/object:Gem::Requirement
|
39
59
|
requirements:
|
40
60
|
- - "~>"
|
41
61
|
- !ruby/object:Gem::Version
|
42
|
-
version: '
|
62
|
+
version: '1.9'
|
43
63
|
- !ruby/object:Gem::Dependency
|
44
64
|
name: sequel
|
45
65
|
requirement: !ruby/object:Gem::Requirement
|
@@ -54,6 +74,20 @@ dependencies:
|
|
54
74
|
- - "~>"
|
55
75
|
- !ruby/object:Gem::Version
|
56
76
|
version: '4.33'
|
77
|
+
- !ruby/object:Gem::Dependency
|
78
|
+
name: redis
|
79
|
+
requirement: !ruby/object:Gem::Requirement
|
80
|
+
requirements:
|
81
|
+
- - "~>"
|
82
|
+
- !ruby/object:Gem::Version
|
83
|
+
version: '3.3'
|
84
|
+
type: :runtime
|
85
|
+
prerelease: false
|
86
|
+
version_requirements: !ruby/object:Gem::Requirement
|
87
|
+
requirements:
|
88
|
+
- - "~>"
|
89
|
+
- !ruby/object:Gem::Version
|
90
|
+
version: '3.3'
|
57
91
|
description:
|
58
92
|
email: r.sprotte@ub.uni-paderborn.de
|
59
93
|
executables:
|
@@ -82,6 +116,10 @@ files:
|
|
82
116
|
- lib/metacrunch/job/buffer.rb
|
83
117
|
- lib/metacrunch/job/dsl.rb
|
84
118
|
- lib/metacrunch/job/dsl/option_support.rb
|
119
|
+
- lib/metacrunch/parallel_processable_reader.rb
|
120
|
+
- lib/metacrunch/redis.rb
|
121
|
+
- lib/metacrunch/redis/queue_reader.rb
|
122
|
+
- lib/metacrunch/redis/queue_writer.rb
|
85
123
|
- lib/metacrunch/test_utils.rb
|
86
124
|
- lib/metacrunch/test_utils/dummy_callable.rb
|
87
125
|
- lib/metacrunch/test_utils/dummy_destination.rb
|