sidekiq-disposal 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +16 -1
- data/README.md +38 -36
- data/lib/sidekiq/disposal/client.rb +33 -9
- data/lib/sidekiq/disposal/server_middleware.rb +7 -4
- data/lib/sidekiq/disposal/version.rb +1 -1
- data/lib/sidekiq/disposal.rb +2 -2
- metadata +2 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 8d1b1751539032559a1b02629ef6e6c71df4a906ade1d44d8951ac18e190d44a
|
4
|
+
data.tar.gz: 3faafb76a82332c5ca86fbfbd3ea3f8363ebfc52c22537cdfc5014a5e0881680
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 5194a09ea875c4a876b92d9e08cf1566365ed5ffeedc6aaee3217182d323614d07b3ca893291038f50fa41960ed445894e97e71ce1754d614cb90ee220847093
|
7
|
+
data.tar.gz: 9d4b2232fce24cf733eb540bbb64280c9aedd4b341eafdd075107e15ba42228a382f3d10ea80ca3ff240ba79eb7f77f1da1e33f7cecc60845ed9d8fa49bcb422
|
data/CHANGELOG.md
CHANGED
@@ -7,4 +7,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
7
7
|
|
8
8
|
## [Unreleased]
|
9
9
|
|
10
|
-
|
10
|
+
## [0.2.0] - 2024-12-19
|
11
|
+
|
12
|
+
### Changed
|
13
|
+
|
14
|
+
- **Breaking**: Replace all usages of "drop" with "discard" to align with Sidekiq terminology ([#4](https://github.com/hibachrach/sidekiq-disposal/pull/4)).
|
15
|
+
- To upgrade:
|
16
|
+
1. Replace all usages of `:drop` with `:discard`
|
17
|
+
1. Replace all usages of `Sidekiq::Disposal::Client::REDIS_DROP_TARGET_SET` with `Sidekiq::Disposal::Client::REDIS_DISCARD_TARGET_SET`
|
18
|
+
1. Replace all usages of `Sidekiq::Disposal::JobDropped` with `Sidekiq::Disposal::JobDiscarded`
|
19
|
+
1. Replace all usages of `Sidekiq::Disposal::Client#drop_target?` with `Sidekiq::Disposal::Client#discard_target?`
|
20
|
+
1. In Redis, execute `COPY sidekiq-disposal:drop_targets sidekiq-disposal:discard_targets`
|
21
|
+
- Eliminate round trip time when calling Redis in Sidekiq middleware ([#3](https://github.com/hibachrach/sidekiq-disposal/pull/3))
|
22
|
+
|
23
|
+
## [0.1.0] - 2024-12-13
|
24
|
+
|
25
|
+
- A new thing! The initial release of `sidekiq-disposal`. 🎉
|
data/README.md
CHANGED
@@ -1,6 +1,8 @@
|
|
1
1
|
# Sidekiq::Disposal
|
2
2
|
|
3
|
-
|
3
|
+
[](https://badge.fury.io/rb/sidekiq-disposal)
|
4
|
+
|
5
|
+
A [Sidekiq][sidekiq] extension to mark Sidekiq Jobs to be disposed of based on the Job ID, Batch ID, or Job Class.
|
4
6
|
Disposal here means to either `:kill` the Job (send to the Dead queue) or `:discard` it (throw it away), at the time the job is picked up and processed by Sidekiq.
|
5
7
|
A disposed Job's `#perform` method will _not_ be called.
|
6
8
|
|
@@ -16,13 +18,13 @@ Or… any number of other ways that some Job, Batch, or Job Class has been enque
|
|
16
18
|
Install the gem and add to the application's Gemfile by executing:
|
17
19
|
|
18
20
|
```console
|
19
|
-
bundle add
|
21
|
+
bundle add sidekiq-disposal
|
20
22
|
```
|
21
23
|
|
22
24
|
If bundler is not being used to manage dependencies, install the gem by executing:
|
23
25
|
|
24
26
|
```console
|
25
|
-
gem install
|
27
|
+
gem install sidekiq-disposal
|
26
28
|
```
|
27
29
|
|
28
30
|
## Usage
|
@@ -37,8 +39,8 @@ client = Sidekiq::Disposal::Client.new
|
|
37
39
|
|
38
40
|
A Job marked to be killed means it will be moved to the Dead queue.
|
39
41
|
|
40
|
-
```
|
41
|
-
# Mark a specific Job
|
42
|
+
```ruby
|
43
|
+
# Mark a specific Job to be killed by specifying its Job ID
|
42
44
|
client.mark(:kill, :jid, some_job_id)
|
43
45
|
|
44
46
|
# Mark a Batch of Jobs to be killed, by Batch ID
|
@@ -50,7 +52,7 @@ client.mark(:kill, :bid, "SomeJobClass")
|
|
50
52
|
|
51
53
|
A Job, Batch, or Job Class can also be `#unmarked` for disposal via a corresponding API.
|
52
54
|
|
53
|
-
```
|
55
|
+
```ruby
|
54
56
|
# Un-mark a specific Job from being killed, by Job ID
|
55
57
|
client.unmark(:kill, :jid, some_job_id)
|
56
58
|
|
@@ -61,50 +63,50 @@ client.unmark(:kill, :bid, some_batch_id)
|
|
61
63
|
client.unmark(:kill, :bid, "SomeJobClass")
|
62
64
|
```
|
63
65
|
|
64
|
-
### Marking to
|
66
|
+
### Marking to Discard
|
65
67
|
|
66
|
-
Similarly, a Job, Batch, or Job Class can be marked to be
|
67
|
-
|
68
|
+
Similarly, a Job, Batch, or Job Class can be marked to be discarded.
|
69
|
+
Discarded jobs are thrown away by Sidekiq - think of them as simply being deleted from the queue, without ever being run.
|
68
70
|
|
69
|
-
```
|
70
|
-
# Mark a specific Job
|
71
|
-
client.mark(:
|
71
|
+
```ruby
|
72
|
+
# Mark a specific Job to be discarded by specifying its Job ID
|
73
|
+
client.mark(:discard, :jid, some_job_id)
|
72
74
|
|
73
|
-
# Mark a Batch of Jobs to be
|
74
|
-
client.mark(:
|
75
|
+
# Mark a Batch of Jobs to be discarded, by Batch ID
|
76
|
+
client.mark(:discard, :bid, some_batch_id)
|
75
77
|
|
76
|
-
# Mark an entire Job Class to be
|
77
|
-
client.mark(:
|
78
|
+
# Mark an entire Job Class to be discarded
|
79
|
+
client.mark(:discard, :bid, "SomeJobClass")
|
78
80
|
```
|
79
81
|
|
80
|
-
And again, there is a corresponding API for un-marking a Job, Batch, or Job Class from being
|
82
|
+
And again, there is a corresponding API for un-marking a Job, Batch, or Job Class from being discarded.
|
81
83
|
|
82
|
-
```
|
83
|
-
# Un-mark a specific Job from being
|
84
|
-
client.unmark(:
|
84
|
+
```ruby
|
85
|
+
# Un-mark a specific Job from being discarded, by Job ID
|
86
|
+
client.unmark(:discard, :jid, some_job_id)
|
85
87
|
|
86
|
-
# Un-mark a Batch of Jobs from being
|
87
|
-
client.unmark(:
|
88
|
+
# Un-mark a Batch of Jobs from being discarded, by Batch ID
|
89
|
+
client.unmark(:discard, :bid, some_batch_id)
|
88
90
|
|
89
|
-
# Un-mark an entire Job Class from being
|
90
|
-
client.unmark(:
|
91
|
+
# Un-mark an entire Job Class from being discarded
|
92
|
+
client.unmark(:discard, :bid, "SomeJobClass")
|
91
93
|
```
|
92
94
|
|
93
95
|
### Un-marking All
|
94
96
|
|
95
|
-
Clearing all `:kill` or `:
|
97
|
+
Clearing all `:kill` or `:discard` marks can be done in one fell swoop as well.
|
96
98
|
|
97
99
|
```ruby
|
98
100
|
client.unmark_all(:kill)
|
99
101
|
|
100
102
|
# or …
|
101
103
|
|
102
|
-
client.unmark_all(:
|
104
|
+
client.unmark_all(:discard)
|
103
105
|
```
|
104
106
|
|
105
107
|
## Configuration
|
106
108
|
|
107
|
-
With `sidekiq-
|
109
|
+
With `sidekiq-disposal` installed, [register its Sidekiq server middleware][sidekiq-register-middleware].
|
108
110
|
Typically this is done via `config/initializers/sidekiq.rb` in a Rails app.
|
109
111
|
|
110
112
|
```ruby
|
@@ -115,24 +117,24 @@ Sidekiq.configure_server do |config|
|
|
115
117
|
end
|
116
118
|
```
|
117
119
|
|
118
|
-
This piece of middleware checks each job, after it's been dequeued, but before
|
119
|
-
If the job is marked for disposal (by Job ID, Batch ID, or Job Class) a corresponding error is raised by the middleware.
|
120
|
+
This piece of middleware checks each job, after it's been dequeued, but before its `#perform` has been called, to see if it should be disposed of.
|
121
|
+
If the job is marked for disposal (by Job ID, Batch ID, or Job Class), a corresponding error is raised by the middleware.
|
120
122
|
|
121
|
-
A Job marked `:kill` will raise a `Sidekiq::Disposal::JobKilled` error, while one marked `:
|
122
|
-
Out of the box these errors will cause [Sidekiq's error handling and retry mechanism][sidekiq-retries] to kick in, re-enqueueing the Job.
|
123
|
+
A Job marked `:kill` will raise a `Sidekiq::Disposal::JobKilled` error, while one marked `:discard` will raise `Sidekiq::Disposal::JobDiscarded`.
|
124
|
+
Out of the box, these errors will cause [Sidekiq's error handling and retry mechanism][sidekiq-retries] to kick in, re-enqueueing the Job.
|
123
125
|
And round-and-round it will go until the default error/death handling kicks in.
|
124
126
|
|
125
127
|
To avoid this, you need to handle those specific `Sidekiq::Disposal` errors accordingly.
|
126
128
|
|
127
|
-
Adjust the base Sidekiq Job class, often called `ApplicationJob` or similar,
|
129
|
+
Adjust the base Sidekiq Job class, often called `ApplicationJob` or similar, so the `sidekiq_retry_in` uses a block similar to this:
|
128
130
|
|
129
131
|
```ruby
|
130
132
|
sidekiq_retry_in do |_count, exception, jobhash|
|
131
133
|
case exception
|
132
|
-
when
|
134
|
+
when Sidekiq::Disposal::JobKilled
|
133
135
|
# Optionally log/collect telemetry here too…
|
134
136
|
:kill
|
135
|
-
when
|
137
|
+
when Sidekiq::Disposal::JobDiscarded
|
136
138
|
# Optionally log/collect telemetry here too…
|
137
139
|
:discard
|
138
140
|
end
|
@@ -142,12 +144,12 @@ end
|
|
142
144
|
_NOTE_: If is not a base job, consider adding one, or you'll need to add this to every job you want to be disposable.
|
143
145
|
|
144
146
|
Returning `:kill` from this method will cause Sidekiq to immediately move the Job to the Dead Queue.
|
145
|
-
Similarly, returning `:discard` will cause Sidekiq to
|
147
|
+
Similarly, returning `:discard` will cause Sidekiq to discard the job on the floor.
|
146
148
|
Either way, the Job's `#perform` is never called.
|
147
149
|
|
148
150
|
### Non-Disposable Jobs
|
149
151
|
|
150
|
-
By default all Jobs are disposable, meaning they _can_ be marked to be `:kill`-ed or `:
|
152
|
+
By default all Jobs are disposable, meaning they _can_ be marked to be `:kill`-ed or `:discard`-ed.
|
151
153
|
However, checking if a specific Job should be disposed of is not free; it requires round trip(s) to Redis.
|
152
154
|
Therefore, you might want to make some Jobs non-disposable to avoid these extra round trips.
|
153
155
|
Or because there are some Jobs that simply should never be disposed of for… _reasons_.
|
@@ -5,7 +5,7 @@ require "sidekiq"
|
|
5
5
|
module Sidekiq
|
6
6
|
module Disposal
|
7
7
|
# A client for marking enqueued jobs for disposal. Disposal can be a job
|
8
|
-
# getting "killed" (sent straight to the dead queue/morgue) or "
|
8
|
+
# getting "killed" (sent straight to the dead queue/morgue) or "discarded"
|
9
9
|
# (hard deleted entirely).
|
10
10
|
#
|
11
11
|
# This task is accomplished with "markers": A job can be "marked" for a
|
@@ -18,7 +18,7 @@ module Sidekiq
|
|
18
18
|
# information).
|
19
19
|
class Client
|
20
20
|
REDIS_KILL_TARGET_SET = "sidekiq-disposal:kill_targets"
|
21
|
-
|
21
|
+
REDIS_DISCARD_TARGET_SET = "sidekiq-disposal:discard_targets"
|
22
22
|
|
23
23
|
ALLOWED_MARKER_TYPES = [
|
24
24
|
:jid,
|
@@ -30,7 +30,7 @@ module Sidekiq
|
|
30
30
|
@sidekiq_api = sidekiq_api
|
31
31
|
end
|
32
32
|
|
33
|
-
# @param disposal_method [:kill, :
|
33
|
+
# @param disposal_method [:kill, :discard] How to handle job
|
34
34
|
# @param marker_type [:jid, :bid, :class_name]
|
35
35
|
# @param marker [String]
|
36
36
|
def mark(disposal_method, marker_type, marker)
|
@@ -41,7 +41,7 @@ module Sidekiq
|
|
41
41
|
end
|
42
42
|
end
|
43
43
|
|
44
|
-
# @param disposal_method [:kill, :
|
44
|
+
# @param disposal_method [:kill, :discard] How to handle job
|
45
45
|
# @param marker_type [:jid, :bid, :class_name]
|
46
46
|
# @param marker [String]
|
47
47
|
def unmark(disposal_method, marker_type, marker)
|
@@ -68,8 +68,32 @@ module Sidekiq
|
|
68
68
|
job_in_target_set?(job, disposal_target_set(:kill))
|
69
69
|
end
|
70
70
|
|
71
|
-
def
|
72
|
-
job_in_target_set?(job, disposal_target_set(:
|
71
|
+
def discard_target?(job)
|
72
|
+
job_in_target_set?(job, disposal_target_set(:discard))
|
73
|
+
end
|
74
|
+
|
75
|
+
# @return [:kill, :discard, nil] Which disposal method the job
|
76
|
+
# is targeted for. If job is not targeted for disposal, `nil`
|
77
|
+
# is returned.
|
78
|
+
def target_disposal_method(job)
|
79
|
+
# We're using `pipelined` as an optimization here to avoid
|
80
|
+
# two separate trips to Redis
|
81
|
+
kill_matches, discard_matches = redis do |conn|
|
82
|
+
conn.pipelined do |pipeline|
|
83
|
+
# `SMISEMBERS setname element1 [element2 ...]` asks whether each
|
84
|
+
# element given is in `setname`; redis-client (the low-level redis
|
85
|
+
# api used by Sidekiq) returns an array of integer answers for
|
86
|
+
# each element: 1 if it's a member, and 0 otherwise.
|
87
|
+
pipeline.smismember(disposal_target_set(:kill), formatted_markers(job))
|
88
|
+
pipeline.smismember(disposal_target_set(:discard), formatted_markers(job))
|
89
|
+
end
|
90
|
+
end
|
91
|
+
|
92
|
+
if kill_matches.any? { |match| match == 1 }
|
93
|
+
:kill
|
94
|
+
elsif discard_matches.any? { |match| match == 1 }
|
95
|
+
:discard
|
96
|
+
end
|
73
97
|
end
|
74
98
|
|
75
99
|
private
|
@@ -82,10 +106,10 @@ module Sidekiq
|
|
82
106
|
case disposal_method
|
83
107
|
when :kill
|
84
108
|
REDIS_KILL_TARGET_SET
|
85
|
-
when :
|
86
|
-
|
109
|
+
when :discard
|
110
|
+
REDIS_DISCARD_TARGET_SET
|
87
111
|
else
|
88
|
-
raise ArgumentError, "disposal_method must be either :kill or :
|
112
|
+
raise ArgumentError, "disposal_method must be either :kill or :discard, instead got: #{disposal_method}"
|
89
113
|
end
|
90
114
|
end
|
91
115
|
|
@@ -15,10 +15,13 @@ module Sidekiq
|
|
15
15
|
def call(job_instance, job, _queue)
|
16
16
|
if job_instance && !job_instance.class.get_sidekiq_options.fetch("disposable", true)
|
17
17
|
yield
|
18
|
-
elsif client.
|
19
|
-
|
20
|
-
|
21
|
-
|
18
|
+
elsif (disposal_method = client.target_disposal_method(job))
|
19
|
+
case disposal_method
|
20
|
+
when :kill
|
21
|
+
raise JobKilled
|
22
|
+
when :discard
|
23
|
+
raise JobDiscarded
|
24
|
+
end
|
22
25
|
else
|
23
26
|
yield
|
24
27
|
end
|
data/lib/sidekiq/disposal.rb
CHANGED
@@ -7,11 +7,11 @@ require_relative "disposal/server_middleware"
|
|
7
7
|
module Sidekiq
|
8
8
|
# Namespace for everything related to job disposal: the process of putting
|
9
9
|
# jobs' markers (i.e. identifying features) on a list so they can be "killed"
|
10
|
-
# (sent immediately to dead set/morgue) or "
|
10
|
+
# (sent immediately to dead set/morgue) or "discarded" (completely discarded
|
11
11
|
# from Sidekiq) when picked up from the queue.
|
12
12
|
module Disposal
|
13
13
|
Error = Class.new(StandardError)
|
14
14
|
JobKilled = Class.new(Error)
|
15
|
-
|
15
|
+
JobDiscarded = Class.new(Error)
|
16
16
|
end
|
17
17
|
end
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: sidekiq-disposal
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.2.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Hazel Bachrach
|
@@ -9,7 +9,7 @@ authors:
|
|
9
9
|
autorequire:
|
10
10
|
bindir: exe
|
11
11
|
cert_chain: []
|
12
|
-
date: 2024-12-
|
12
|
+
date: 2024-12-19 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: sidekiq
|