sidekiq-disposal 0.1.0 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 8ecd41e2b17b1168b119bf60002c1680c960561f0f4c1d972648a08386ed1423
4
- data.tar.gz: 14df2e7593177e8c2fb90d96d1acbe3ed57206987cb59763e06fdf9db4a7ba63
3
+ metadata.gz: 8d1b1751539032559a1b02629ef6e6c71df4a906ade1d44d8951ac18e190d44a
4
+ data.tar.gz: 3faafb76a82332c5ca86fbfbd3ea3f8363ebfc52c22537cdfc5014a5e0881680
5
5
  SHA512:
6
- metadata.gz: 9e5e2a2084643fe72cafc7cbe7eb44dfacf500a63744d75585a6a38f63f8bf853090589183de163e487fbbeb975cd3e8e3ee8f11459c5b3e03a0ef890da2770e
7
- data.tar.gz: 7bc4001093d88ef742f20102b5b11d98087726e9b076a7e733c8f695c200799a9c11282ef7c43d6b5d7de05fbd6a879ff2c83638e79ab86eaecc90b3e1aec24f
6
+ metadata.gz: 5194a09ea875c4a876b92d9e08cf1566365ed5ffeedc6aaee3217182d323614d07b3ca893291038f50fa41960ed445894e97e71ce1754d614cb90ee220847093
7
+ data.tar.gz: 9d4b2232fce24cf733eb540bbb64280c9aedd4b341eafdd075107e15ba42228a382f3d10ea80ca3ff240ba79eb7f77f1da1e33f7cecc60845ed9d8fa49bcb422
data/CHANGELOG.md CHANGED
@@ -7,4 +7,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
7
7
 
8
8
  ## [Unreleased]
9
9
 
10
- - A new thing!
10
+ ## [0.2.0] - 2024-12-19
11
+
12
+ ### Changed
13
+
14
+ - **Breaking**: Replace all usages of "drop" with "discard" to align with Sidekiq terminology ([#4](https://github.com/hibachrach/sidekiq-disposal/pull/4)).
15
+ - To upgrade:
16
+ 1. Replace all usages of `:drop` with `:discard`
17
+ 1. Replace all usages of `Sidekiq::Disposal::Client::REDIS_DROP_TARGET_SET` with `Sidekiq::Disposal::Client::REDIS_DISCARD_TARGET_SET`
18
+ 1. Replace all usages of `Sidekiq::Disposal::JobDropped` with `Sidekiq::Disposal::JobDiscarded`
19
+ 1. Replace all usages of `Sidekiq::Disposal::Client#drop_target?` with `Sidekiq::Disposal::Client#discard_target?`
20
+ 1. In Redis, execute `COPY sidekiq-disposal:drop_targets sidekiq-disposal:discard_targets`
21
+ - Eliminate round trip time when calling Redis in Sidekiq middleware ([#3](https://github.com/hibachrach/sidekiq-disposal/pull/3))
22
+
23
+ ## [0.1.0] - 2024-12-13
24
+
25
+ - A new thing! The initial release of `sidekiq-disposal`. 🎉
data/README.md CHANGED
@@ -1,6 +1,8 @@
1
1
  # Sidekiq::Disposal
2
2
 
3
- A [Sidekiq][sidekiq] extension mark Sidekiq Jobs to be disposed of based on the Job ID, Batch ID, or Job Class.
3
+ [![Gem Version](https://badge.fury.io/rb/sidekiq-disposal.svg?icon=si%3Arubygems&icon_color=%23ff2600)](https://badge.fury.io/rb/sidekiq-disposal)
4
+
5
+ A [Sidekiq][sidekiq] extension to mark Sidekiq Jobs to be disposed of based on the Job ID, Batch ID, or Job Class.
4
6
  Disposal here means to either `:kill` the Job (send to the Dead queue) or `:discard` it (throw it away), at the time the job is picked up and processed by Sidekiq.
5
7
  A disposed Job's `#perform` method will _not_ be called.
6
8
 
@@ -16,13 +18,13 @@ Or… any number of other ways that some Job, Batch, or Job Class has been enque
16
18
  Install the gem and add to the application's Gemfile by executing:
17
19
 
18
20
  ```console
19
- bundle add UPDATE_WITH_YOUR_GEM_NAME_IMMEDIATELY_AFTER_RELEASE_TO_RUBYGEMS_ORG
21
+ bundle add sidekiq-disposal
20
22
  ```
21
23
 
22
24
  If bundler is not being used to manage dependencies, install the gem by executing:
23
25
 
24
26
  ```console
25
- gem install UPDATE_WITH_YOUR_GEM_NAME_IMMEDIATELY_AFTER_RELEASE_TO_RUBYGEMS_ORG
27
+ gem install sidekiq-disposal
26
28
  ```
27
29
 
28
30
  ## Usage
@@ -37,8 +39,8 @@ client = Sidekiq::Disposal::Client.new
37
39
 
38
40
  A Job marked to be killed means it will be moved to the Dead queue.
39
41
 
40
- ```
41
- # Mark a specific Job, by its ID to be killed
42
+ ```ruby
43
+ # Mark a specific Job to be killed by specifying its Job ID
42
44
  client.mark(:kill, :jid, some_job_id)
43
45
 
44
46
  # Mark a Batch of Jobs to be killed, by Batch ID
@@ -50,7 +52,7 @@ client.mark(:kill, :bid, "SomeJobClass")
50
52
 
51
53
  A Job, Batch, or Job Class can also be `#unmarked` for disposal via a corresponding API.
52
54
 
53
- ```
55
+ ```ruby
54
56
  # Un-mark a specific Job from being killed, by Job ID
55
57
  client.unmark(:kill, :jid, some_job_id)
56
58
 
@@ -61,50 +63,50 @@ client.unmark(:kill, :bid, some_batch_id)
61
63
  client.unmark(:kill, :bid, "SomeJobClass")
62
64
  ```
63
65
 
64
- ### Marking to Drop
66
+ ### Marking to Discard
65
67
 
66
- Similarly, a Job, Batch, or Job Class can be marked to be dropped.
67
- Dropped jobs are discarded by Sidekiq - think of them as simply being deleted from the queue, without ever being run.
68
+ Similarly, a Job, Batch, or Job Class can be marked to be discarded.
69
+ Discarded jobs are thrown away by Sidekiq - think of them as simply being deleted from the queue, without ever being run.
68
70
 
69
- ```
70
- # Mark a specific Job, by its ID to be dropped
71
- client.mark(:drop, :jid, some_job_id)
71
+ ```ruby
72
+ # Mark a specific Job to be discarded by specifying its Job ID
73
+ client.mark(:discard, :jid, some_job_id)
72
74
 
73
- # Mark a Batch of Jobs to be dropped, by Batch ID
74
- client.mark(:drop, :bid, some_batch_id)
75
+ # Mark a Batch of Jobs to be discarded, by Batch ID
76
+ client.mark(:discard, :bid, some_batch_id)
75
77
 
76
- # Mark an entire Job Class to be dropped
77
- client.mark(:drop, :bid, "SomeJobClass")
78
+ # Mark an entire Job Class to be discarded
79
+ client.mark(:discard, :bid, "SomeJobClass")
78
80
  ```
79
81
 
80
- And again, there is a corresponding API for un-marking a Job, Batch, or Job Class from being dropped.
82
+ And again, there is a corresponding API for un-marking a Job, Batch, or Job Class from being discarded.
81
83
 
82
- ```
83
- # Un-mark a specific Job from being dropped, by Job ID
84
- client.unmark(:drop, :jid, some_job_id)
84
+ ```ruby
85
+ # Un-mark a specific Job from being discarded, by Job ID
86
+ client.unmark(:discard, :jid, some_job_id)
85
87
 
86
- # Un-mark a Batch of Jobs from being dropped, by Batch ID
87
- client.unmark(:drop, :bid, some_batch_id)
88
+ # Un-mark a Batch of Jobs from being discarded, by Batch ID
89
+ client.unmark(:discard, :bid, some_batch_id)
88
90
 
89
- # Un-mark an entire Job Class from being dropped
90
- client.unmark(:drop, :bid, "SomeJobClass")
91
+ # Un-mark an entire Job Class from being discarded
92
+ client.unmark(:discard, :bid, "SomeJobClass")
91
93
  ```
92
94
 
93
95
  ### Un-marking All
94
96
 
95
- Clearing all `:kill` or `:drop` marks can be done in one fell swoop as well.
97
+ Clearing all `:kill` or `:discard` marks can be done in one fell swoop as well.
96
98
 
97
99
  ```ruby
98
100
  client.unmark_all(:kill)
99
101
 
100
102
  # or …
101
103
 
102
- client.unmark_all(:drop)
104
+ client.unmark_all(:discard)
103
105
  ```
104
106
 
105
107
  ## Configuration
106
108
 
107
- With `sidekiq-dispose` installed, [register its Sidekiq server middleware][sidekiq-register-middleware].
109
+ With `sidekiq-disposal` installed, [register its Sidekiq server middleware][sidekiq-register-middleware].
108
110
  Typically this is done via `config/initializers/sidekiq.rb` in a Rails app.
109
111
 
110
112
  ```ruby
@@ -115,24 +117,24 @@ Sidekiq.configure_server do |config|
115
117
  end
116
118
  ```
117
119
 
118
- This piece of middleware checks each job, after it's been dequeued, but before it's `#perform` has been called, to see if it should be disposed of.
119
- If the job is marked for disposal (by Job ID, Batch ID, or Job Class) a corresponding error is raised by the middleware.
120
+ This piece of middleware checks each job, after it's been dequeued, but before its `#perform` has been called, to see if it should be disposed of.
121
+ If the job is marked for disposal (by Job ID, Batch ID, or Job Class), a corresponding error is raised by the middleware.
120
122
 
121
- A Job marked `:kill` will raise a `Sidekiq::Disposal::JobKilled` error, while one marked `:drop` will raise `Sidekiq::Disposal::JobDropped`.
122
- Out of the box these errors will cause [Sidekiq's error handling and retry mechanism][sidekiq-retries] to kick in, re-enqueueing the Job.
123
+ A Job marked `:kill` will raise a `Sidekiq::Disposal::JobKilled` error, while one marked `:discard` will raise `Sidekiq::Disposal::JobDiscarded`.
124
+ Out of the box, these errors will cause [Sidekiq's error handling and retry mechanism][sidekiq-retries] to kick in, re-enqueueing the Job.
123
125
  And round-and-round it will go until the default error/death handling kicks in.
124
126
 
125
127
  To avoid this, you need to handle those specific `Sidekiq::Disposal` errors accordingly.
126
128
 
127
- Adjust the base Sidekiq Job class, often called `ApplicationJob` or similar, to overwrite or tweak the `#sidekiq_retry_in` method:
129
+ Adjust the base Sidekiq Job class, often called `ApplicationJob` or similar, so the `sidekiq_retry_in` uses a block similar to this:
128
130
 
129
131
  ```ruby
130
132
  sidekiq_retry_in do |_count, exception, jobhash|
131
133
  case exception
132
- when Infrastructure::Sidekiq::Disposal::JobKilled
134
+ when Sidekiq::Disposal::JobKilled
133
135
  # Optionally log/collect telemetry here too…
134
136
  :kill
135
- when Infrastructure::Sidekiq::Disposal::JobDropped
137
+ when Sidekiq::Disposal::JobDiscarded
136
138
  # Optionally log/collect telemetry here too…
137
139
  :discard
138
140
  end
@@ -142,12 +144,12 @@ end
142
144
  _NOTE_: If is not a base job, consider adding one, or you'll need to add this to every job you want to be disposable.
143
145
 
144
146
  Returning `:kill` from this method will cause Sidekiq to immediately move the Job to the Dead Queue.
145
- Similarly, returning `:discard` will cause Sidekiq to drop the job on the floor.
147
+ Similarly, returning `:discard` will cause Sidekiq to discard the job on the floor.
146
148
  Either way, the Job's `#perform` is never called.
147
149
 
148
150
  ### Non-Disposable Jobs
149
151
 
150
- By default all Jobs are disposable, meaning they _can_ be marked to be `:kill`-ed or `:drop`-ed.
152
+ By default all Jobs are disposable, meaning they _can_ be marked to be `:kill`-ed or `:discard`-ed.
151
153
  However, checking if a specific Job should be disposed of is not free; it requires round trip(s) to Redis.
152
154
  Therefore, you might want to make some Jobs non-disposable to avoid these extra round trips.
153
155
  Or because there are some Jobs that simply should never be disposed of for… _reasons_.
@@ -5,7 +5,7 @@ require "sidekiq"
5
5
  module Sidekiq
6
6
  module Disposal
7
7
  # A client for marking enqueued jobs for disposal. Disposal can be a job
8
- # getting "killed" (sent straight to the dead queue/morgue) or "dropped"
8
+ # getting "killed" (sent straight to the dead queue/morgue) or "discarded"
9
9
  # (hard deleted entirely).
10
10
  #
11
11
  # This task is accomplished with "markers": A job can be "marked" for a
@@ -18,7 +18,7 @@ module Sidekiq
18
18
  # information).
19
19
  class Client
20
20
  REDIS_KILL_TARGET_SET = "sidekiq-disposal:kill_targets"
21
- REDIS_DROP_TARGET_SET = "sidekiq-disposal:drop_targets"
21
+ REDIS_DISCARD_TARGET_SET = "sidekiq-disposal:discard_targets"
22
22
 
23
23
  ALLOWED_MARKER_TYPES = [
24
24
  :jid,
@@ -30,7 +30,7 @@ module Sidekiq
30
30
  @sidekiq_api = sidekiq_api
31
31
  end
32
32
 
33
- # @param disposal_method [:kill, :drop] How to handle job
33
+ # @param disposal_method [:kill, :discard] How to handle job
34
34
  # @param marker_type [:jid, :bid, :class_name]
35
35
  # @param marker [String]
36
36
  def mark(disposal_method, marker_type, marker)
@@ -41,7 +41,7 @@ module Sidekiq
41
41
  end
42
42
  end
43
43
 
44
- # @param disposal_method [:kill, :drop] How to handle job
44
+ # @param disposal_method [:kill, :discard] How to handle job
45
45
  # @param marker_type [:jid, :bid, :class_name]
46
46
  # @param marker [String]
47
47
  def unmark(disposal_method, marker_type, marker)
@@ -68,8 +68,32 @@ module Sidekiq
68
68
  job_in_target_set?(job, disposal_target_set(:kill))
69
69
  end
70
70
 
71
- def drop_target?(job)
72
- job_in_target_set?(job, disposal_target_set(:drop))
71
+ def discard_target?(job)
72
+ job_in_target_set?(job, disposal_target_set(:discard))
73
+ end
74
+
75
+ # @return [:kill, :discard, nil] Which disposal method the job
76
+ # is targeted for. If job is not targeted for disposal, `nil`
77
+ # is returned.
78
+ def target_disposal_method(job)
79
+ # We're using `pipelined` as an optimization here to avoid
80
+ # two separate trips to Redis
81
+ kill_matches, discard_matches = redis do |conn|
82
+ conn.pipelined do |pipeline|
83
+ # `SMISEMBERS setname element1 [element2 ...]` asks whether each
84
+ # element given is in `setname`; redis-client (the low-level redis
85
+ # api used by Sidekiq) returns an array of integer answers for
86
+ # each element: 1 if it's a member, and 0 otherwise.
87
+ pipeline.smismember(disposal_target_set(:kill), formatted_markers(job))
88
+ pipeline.smismember(disposal_target_set(:discard), formatted_markers(job))
89
+ end
90
+ end
91
+
92
+ if kill_matches.any? { |match| match == 1 }
93
+ :kill
94
+ elsif discard_matches.any? { |match| match == 1 }
95
+ :discard
96
+ end
73
97
  end
74
98
 
75
99
  private
@@ -82,10 +106,10 @@ module Sidekiq
82
106
  case disposal_method
83
107
  when :kill
84
108
  REDIS_KILL_TARGET_SET
85
- when :drop
86
- REDIS_DROP_TARGET_SET
109
+ when :discard
110
+ REDIS_DISCARD_TARGET_SET
87
111
  else
88
- raise ArgumentError, "disposal_method must be either :kill or :drop, instead got: #{disposal_method}"
112
+ raise ArgumentError, "disposal_method must be either :kill or :discard, instead got: #{disposal_method}"
89
113
  end
90
114
  end
91
115
 
@@ -15,10 +15,13 @@ module Sidekiq
15
15
  def call(job_instance, job, _queue)
16
16
  if job_instance && !job_instance.class.get_sidekiq_options.fetch("disposable", true)
17
17
  yield
18
- elsif client.kill_target?(job)
19
- raise JobKilled
20
- elsif client.drop_target?(job)
21
- raise JobDropped
18
+ elsif (disposal_method = client.target_disposal_method(job))
19
+ case disposal_method
20
+ when :kill
21
+ raise JobKilled
22
+ when :discard
23
+ raise JobDiscarded
24
+ end
22
25
  else
23
26
  yield
24
27
  end
@@ -2,6 +2,6 @@
2
2
 
3
3
  module Sidekiq
4
4
  module Disposal
5
- VERSION = "0.1.0"
5
+ VERSION = "0.2.0"
6
6
  end
7
7
  end
@@ -7,11 +7,11 @@ require_relative "disposal/server_middleware"
7
7
  module Sidekiq
8
8
  # Namespace for everything related to job disposal: the process of putting
9
9
  # jobs' markers (i.e. identifying features) on a list so they can be "killed"
10
- # (sent immediately to dead set/morgue) or "dropped" (completely discarded
10
+ # (sent immediately to dead set/morgue) or "discarded" (completely discarded
11
11
  # from Sidekiq) when picked up from the queue.
12
12
  module Disposal
13
13
  Error = Class.new(StandardError)
14
14
  JobKilled = Class.new(Error)
15
- JobDropped = Class.new(Error)
15
+ JobDiscarded = Class.new(Error)
16
16
  end
17
17
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: sidekiq-disposal
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.0
4
+ version: 0.2.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Hazel Bachrach
@@ -9,7 +9,7 @@ authors:
9
9
  autorequire:
10
10
  bindir: exe
11
11
  cert_chain: []
12
- date: 2024-12-13 00:00:00.000000000 Z
12
+ date: 2024-12-19 00:00:00.000000000 Z
13
13
  dependencies:
14
14
  - !ruby/object:Gem::Dependency
15
15
  name: sidekiq