jobba 1.1.0 → 1.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +70 -26
- data/lib/jobba/clause.rb +1 -1
- data/lib/jobba/clause_factory.rb +2 -0
- data/lib/jobba/exceptions.rb +1 -0
- data/lib/jobba/id_clause.rb +14 -0
- data/lib/jobba/query.rb +68 -26
- data/lib/jobba/statuses.rb +29 -8
- data/lib/jobba/utils.rb +4 -0
- data/lib/jobba/version.rb +1 -1
- data/lib/jobba.rb +3 -7
- metadata +3 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: dcfba40866333af0570df573302aaac92ef2c985
|
4
|
+
data.tar.gz: 031ee23b4e6ef49a80c46f539e68b7772a621edc
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: ddd79547cd5913d6b7c10962150ebec677055ab979ad0890685d2362fd0123cd312595efeb27ad8a39618ef4cc256cc99498bccfa889f95a6f8452db42a62fc0
|
7
|
+
data.tar.gz: e9442849d2a665b3b0253d5f1ada4ac74f61441e7976b3a40d058e758f72bef9c80c6fc467caea97c3fa590f1f3cf6d7f05f9c6f3188418bb8410dae2f0cbce5
|
data/README.md
CHANGED
@@ -19,7 +19,7 @@ or
|
|
19
19
|
$> gem install jobba
|
20
20
|
```
|
21
21
|
|
22
|
-
|
22
|
+
Version 1.x.x follows the scheme, 1.major_change.minor_change. Normal semantic versioning (major/minor/patch) will begin with version `2.0.0`.
|
23
23
|
|
24
24
|
## Configuration
|
25
25
|
|
@@ -237,7 +237,7 @@ You can also call `reload!` on a `Status` to have it reset its state to what is
|
|
237
237
|
Once jobs are completed or otherwise no longer interesting, it'd be nice to clear them out of Redis. You can do this with:
|
238
238
|
|
239
239
|
```ruby
|
240
|
-
my_status.delete # freaks out if `my_status` isn't
|
240
|
+
my_status.delete # freaks out if `my_status` isn't completed
|
241
241
|
my_status.delete! # always deletes
|
242
242
|
```
|
243
243
|
|
@@ -247,6 +247,12 @@ Jobba has an activerecord-like query interface for finding Status objects.
|
|
247
247
|
|
248
248
|
### Basic Query Examples
|
249
249
|
|
250
|
+
**Getting All Statuses**
|
251
|
+
|
252
|
+
```ruby
|
253
|
+
Jobba.all
|
254
|
+
```
|
255
|
+
|
250
256
|
**State**
|
251
257
|
|
252
258
|
```ruby
|
@@ -305,6 +311,15 @@ Jobba.where(job_arg: "gid://app/MyModel/42")
|
|
305
311
|
Jobba.where(job_arg: "gid://app/Person/86")
|
306
312
|
```
|
307
313
|
|
314
|
+
**Status IDs**
|
315
|
+
|
316
|
+
```ruby
|
317
|
+
Jobba.where(id: nil)
|
318
|
+
Jobba.where(id: [])
|
319
|
+
Jobba.where(id: "some_id")
|
320
|
+
Jobba.where(id: ["an_id", "another_id"])
|
321
|
+
```
|
322
|
+
|
308
323
|
### Query Chaining
|
309
324
|
|
310
325
|
Queries can be chained! (intersects the results of each `where` clause)
|
@@ -314,47 +329,79 @@ Jobba.where(state: :queued).where(recorded_at: {after: some_time})
|
|
314
329
|
Jobba.where(job_name: "MyTroublesomeJob").where(state: :failed)
|
315
330
|
```
|
316
331
|
|
317
|
-
###
|
332
|
+
### Sort Order
|
333
|
+
|
334
|
+
Currently, results from queries are not guaranteed to be in any order. You can sort them yourself using normal Ruby calls.
|
335
|
+
|
336
|
+
### Running a Query to get Statuses
|
337
|
+
|
338
|
+
```ruby
|
339
|
+
Jobba.where(...).run
|
340
|
+
```
|
341
|
+
|
342
|
+
When you call `run` on a query, you'll get back a `Statuses` object, which is simply a collection of `Status` objects with a few convenience methods and bulk operations.
|
343
|
+
|
344
|
+
**Bulk Methods on Statuses**
|
345
|
+
|
346
|
+
* `delete_all`
|
347
|
+
* `delete_all!`
|
348
|
+
* `request_kill_all!`
|
349
|
+
|
350
|
+
These work like describe above for individual `Status` objects.
|
351
|
+
|
352
|
+
There is also a not-very-tested `multi` operation that takes a block and executes the block inside a Redis `multi` call. Do not use it unless you really know what you are doing.
|
318
353
|
|
319
|
-
|
354
|
+
```ruby
|
355
|
+
my_statuses.multi do |status, redis|
|
356
|
+
# do stuff on `status` using the `redis` connection
|
357
|
+
end
|
358
|
+
```
|
359
|
+
|
360
|
+
**Array-like Methods on Statuses**
|
320
361
|
|
321
|
-
* `first`
|
322
362
|
* `any?`
|
323
363
|
* `none?`
|
324
364
|
* `all?`
|
325
|
-
* `each`
|
326
|
-
* `each_with_index`
|
327
365
|
* `map`
|
328
366
|
* `collect`
|
329
|
-
* `select`
|
330
367
|
* `empty?`
|
331
368
|
* `count`
|
369
|
+
* `select!`
|
370
|
+
* `reject!`
|
332
371
|
|
333
|
-
|
334
|
-
|
335
|
-
You can also call two special methods directly on `Jobba`:
|
372
|
+
If you want to get an array of `Status` objects from a `Statuses` object, just call
|
336
373
|
|
337
374
|
```ruby
|
338
|
-
|
339
|
-
Jobba.count # returns count of all statuses
|
375
|
+
a_statuses_object.to_a
|
340
376
|
```
|
341
377
|
|
342
|
-
|
378
|
+
`select!` and `reject!`, as you would expect, operate in place and also return `self`.
|
343
379
|
|
344
|
-
|
380
|
+
### Passthrough Methods on Queries
|
345
381
|
|
346
|
-
|
347
|
-
* `delete!`
|
348
|
-
* `request_kill!`
|
382
|
+
As a convenience, if you call a method on `Query` that isn't defined there but is defined on `Statuses`, a new `Statuses` object will be created for you and your method called on it.
|
349
383
|
|
350
|
-
|
384
|
+
```ruby
|
385
|
+
Jobba.where(state: :queued).collect(&:queued_at)
|
386
|
+
```
|
351
387
|
|
352
|
-
|
388
|
+
is the same as
|
353
389
|
|
354
390
|
```ruby
|
355
|
-
|
356
|
-
|
357
|
-
|
391
|
+
Jobba.where(state: :queued).run.collect(&:queued_at)
|
392
|
+
```
|
393
|
+
|
394
|
+
### Query Counts
|
395
|
+
|
396
|
+
Notably, both `Query` and `Statuses` define the `count` and `empty?` methods. Which ones you use affects if the counting is done in Redis or in Ruby:
|
397
|
+
|
398
|
+
```ruby
|
399
|
+
Jobba.where(...).count # These count in Redis
|
400
|
+
Jobba.where(...).empty?
|
401
|
+
Jobba.all.count
|
402
|
+
|
403
|
+
Jobba.where(...).run.count # These pull data back to Ruby and count in Ruby
|
404
|
+
Jobba.where(...).run.empty?
|
358
405
|
```
|
359
406
|
|
360
407
|
## Notes
|
@@ -372,9 +419,6 @@ Jobba strives to do all of its operations as efficiently as possible using built
|
|
372
419
|
1. Provide job min, max, and average durations.
|
373
420
|
2. Implement `add_error`.
|
374
421
|
8. Specs that test scale.
|
375
|
-
9. Make sure we're calling `multi` or `pipelined` everywhere we can.
|
376
|
-
11. Make sure we're consistent on completed/complete incompleted/incomplete.
|
377
|
-
12. Should more `Statuses` operations return `Statuses`, e.g. `each`, so that they can be chained?
|
378
422
|
|
379
423
|
|
380
424
|
|
data/lib/jobba/clause.rb
CHANGED
@@ -25,7 +25,7 @@ class Jobba::Clause
|
|
25
25
|
end
|
26
26
|
|
27
27
|
def to_new_set
|
28
|
-
new_key =
|
28
|
+
new_key = Jobba::Utils.temp_key
|
29
29
|
|
30
30
|
# Make a copy of the data into new_key then filter values if indicated
|
31
31
|
# (always making a copy gets normal sets into a sorted set key OR if
|
data/lib/jobba/clause_factory.rb
CHANGED
data/lib/jobba/exceptions.rb
CHANGED
data/lib/jobba/query.rb
CHANGED
@@ -1,4 +1,5 @@
|
|
1
1
|
require 'jobba/clause'
|
2
|
+
require 'jobba/id_clause'
|
2
3
|
require 'jobba/clause_factory'
|
3
4
|
|
4
5
|
class Jobba::Query
|
@@ -14,7 +15,7 @@ class Jobba::Query
|
|
14
15
|
end
|
15
16
|
|
16
17
|
def count
|
17
|
-
|
18
|
+
_run(COUNT_STATUSES)
|
18
19
|
end
|
19
20
|
|
20
21
|
def empty?
|
@@ -25,9 +26,9 @@ class Jobba::Query
|
|
25
26
|
# to run on the result of the executed `where`s. So if we don't know what
|
26
27
|
# the method is, execute the `where`s and pass the method to its output.
|
27
28
|
|
28
|
-
def method_missing(method_name, *args)
|
29
|
+
def method_missing(method_name, *args, &block)
|
29
30
|
if Jobba::Statuses.instance_methods.include?(method_name)
|
30
|
-
run
|
31
|
+
run.send(method_name, *args, &block)
|
31
32
|
else
|
32
33
|
super
|
33
34
|
end
|
@@ -37,6 +38,10 @@ class Jobba::Query
|
|
37
38
|
Jobba::Statuses.instance_methods.include?(method_name) || super
|
38
39
|
end
|
39
40
|
|
41
|
+
def run
|
42
|
+
_run(GET_STATUSES)
|
43
|
+
end
|
44
|
+
|
40
45
|
protected
|
41
46
|
|
42
47
|
attr_accessor :clauses
|
@@ -45,36 +50,73 @@ class Jobba::Query
|
|
45
50
|
@clauses = []
|
46
51
|
end
|
47
52
|
|
48
|
-
|
49
|
-
|
50
|
-
Jobba::Statuses.new(ids)
|
51
|
-
}
|
52
|
-
|
53
|
-
COUNT_STATUSES = ->(working_set) {
|
54
|
-
Jobba.redis.zcard(working_set)
|
55
|
-
}
|
53
|
+
class RunBlocks
|
54
|
+
attr_reader :redis_block, :output_block
|
56
55
|
|
57
|
-
|
58
|
-
|
59
|
-
|
60
|
-
|
61
|
-
load_default_clause if clauses.empty?
|
62
|
-
working_set = nil
|
56
|
+
def initialize(redis_block, output_block)
|
57
|
+
@redis_block = redis_block
|
58
|
+
@output_block = output_block
|
59
|
+
end
|
63
60
|
|
64
|
-
|
65
|
-
|
61
|
+
def output_block_result_is_redis_block_result?
|
62
|
+
output_block.nil?
|
63
|
+
end
|
64
|
+
end
|
66
65
|
|
67
|
-
|
68
|
-
|
69
|
-
|
70
|
-
|
71
|
-
|
66
|
+
GET_STATUSES = RunBlocks.new(
|
67
|
+
->(working_set, redis) {
|
68
|
+
redis.zrange(working_set, 0, -1)
|
69
|
+
},
|
70
|
+
->(ids) {
|
71
|
+
Jobba::Statuses.new(ids)
|
72
|
+
}
|
73
|
+
)
|
74
|
+
|
75
|
+
COUNT_STATUSES = RunBlocks.new(
|
76
|
+
->(working_set, redis) {
|
77
|
+
redis.zcard(working_set)
|
78
|
+
},
|
79
|
+
nil
|
80
|
+
)
|
81
|
+
|
82
|
+
def _run(run_blocks)
|
83
|
+
# Each clause in a query is converted to a sorted set (which may be filtered,
|
84
|
+
# e.g. in the case of timestamp clauses) and then the sets are successively
|
85
|
+
# intersected.
|
86
|
+
#
|
87
|
+
# Different users of this method have different uses for the final "working"
|
88
|
+
# set. Because we want to bundle all of the creations and intersections of
|
89
|
+
# clause sets into one call to Redis (via a `multi` block), we have users
|
90
|
+
# of this method provide a final block to run on the working set within
|
91
|
+
# Redis (and within the `multi` call) and then another block to run on
|
92
|
+
# the output of the first block.
|
93
|
+
|
94
|
+
multi_result = redis.multi do
|
95
|
+
load_default_clause if clauses.empty?
|
96
|
+
working_set = nil
|
97
|
+
|
98
|
+
clauses.each_with_index do |clause, ii|
|
99
|
+
clause_set = clause.to_new_set
|
100
|
+
|
101
|
+
if working_set.nil?
|
102
|
+
working_set = clause_set
|
103
|
+
else
|
104
|
+
redis.zinterstore(working_set, [working_set, clause_set], weights: [0, 0])
|
105
|
+
redis.del(clause_set)
|
106
|
+
end
|
72
107
|
end
|
73
|
-
end
|
74
108
|
|
75
|
-
|
109
|
+
# This is later accessed as `multi_result[-2]` since it is the second to last output
|
110
|
+
run_blocks.redis_block.call(working_set, redis)
|
111
|
+
|
76
112
|
redis.del(working_set)
|
77
113
|
end
|
114
|
+
|
115
|
+
redis_block_output = multi_result[-2]
|
116
|
+
|
117
|
+
run_blocks.output_block_result_is_redis_block_result? ?
|
118
|
+
redis_block_output :
|
119
|
+
run_blocks.output_block.call(redis_block_output)
|
78
120
|
end
|
79
121
|
|
80
122
|
def load_default_clause
|
data/lib/jobba/statuses.rb
CHANGED
@@ -5,15 +5,23 @@ class Jobba::Statuses
|
|
5
5
|
|
6
6
|
attr_reader :ids
|
7
7
|
|
8
|
-
def
|
9
|
-
load
|
8
|
+
def to_a
|
9
|
+
load.dup
|
10
|
+
end
|
11
|
+
|
12
|
+
def_delegators :@ids, :empty?
|
13
|
+
|
14
|
+
def_delegators :load, :any?, :none?, :all?, :count, :map, :collect
|
15
|
+
|
16
|
+
def select!(&block)
|
17
|
+
modify!(:select!, &block)
|
10
18
|
end
|
11
19
|
|
12
|
-
|
13
|
-
|
14
|
-
|
20
|
+
def reject!(&block)
|
21
|
+
modify!(:reject!, &block)
|
22
|
+
end
|
15
23
|
|
16
|
-
def
|
24
|
+
def delete_all
|
17
25
|
if any?(&:incomplete?)
|
18
26
|
raise(Jobba::NotCompletedError,
|
19
27
|
"This status cannot be deleted because it isn't complete. Use " \
|
@@ -23,7 +31,7 @@ class Jobba::Statuses
|
|
23
31
|
delete!
|
24
32
|
end
|
25
33
|
|
26
|
-
def
|
34
|
+
def delete_all!
|
27
35
|
load
|
28
36
|
redis.multi do
|
29
37
|
@cache.each(&:delete!)
|
@@ -32,7 +40,7 @@ class Jobba::Statuses
|
|
32
40
|
@ids = []
|
33
41
|
end
|
34
42
|
|
35
|
-
def
|
43
|
+
def request_kill_all!
|
36
44
|
load
|
37
45
|
redis.multi do
|
38
46
|
@cache.each(&:request_kill!)
|
@@ -61,11 +69,24 @@ class Jobba::Statuses
|
|
61
69
|
end
|
62
70
|
end
|
63
71
|
|
72
|
+
raw_statuses.reject!(&:empty?)
|
73
|
+
|
64
74
|
raw_statuses.collect do |raw_status|
|
65
75
|
Jobba::Status.new(raw: raw_status)
|
66
76
|
end
|
67
77
|
end
|
68
78
|
|
79
|
+
def modify!(method, &block)
|
80
|
+
raise Jobba::NotImplemented unless block_given?
|
81
|
+
load
|
82
|
+
if @cache.send(method, &block).nil?
|
83
|
+
nil
|
84
|
+
else
|
85
|
+
@ids = @cache.collect(&:id)
|
86
|
+
self
|
87
|
+
end
|
88
|
+
end
|
89
|
+
|
69
90
|
def initialize(*ids)
|
70
91
|
@ids = [ids].flatten.compact
|
71
92
|
@cache = nil
|
data/lib/jobba/utils.rb
CHANGED
data/lib/jobba/version.rb
CHANGED
data/lib/jobba.rb
CHANGED
@@ -14,16 +14,12 @@ require "jobba/query"
|
|
14
14
|
|
15
15
|
module Jobba
|
16
16
|
|
17
|
-
def self.where(*args)
|
18
|
-
Query.new.where(*args)
|
19
|
-
end
|
20
|
-
|
21
17
|
def self.all
|
22
|
-
Query.new
|
18
|
+
Query.new
|
23
19
|
end
|
24
20
|
|
25
|
-
def self.
|
26
|
-
|
21
|
+
def self.where(*args)
|
22
|
+
all.where(*args)
|
27
23
|
end
|
28
24
|
|
29
25
|
def self.configure
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: jobba
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.
|
4
|
+
version: 1.2.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- JP Slavinsky
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2016-01-
|
11
|
+
date: 2016-01-03 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: redis
|
@@ -133,6 +133,7 @@ files:
|
|
133
133
|
- lib/jobba/common.rb
|
134
134
|
- lib/jobba/configuration.rb
|
135
135
|
- lib/jobba/exceptions.rb
|
136
|
+
- lib/jobba/id_clause.rb
|
136
137
|
- lib/jobba/query.rb
|
137
138
|
- lib/jobba/state.rb
|
138
139
|
- lib/jobba/status.rb
|