sidekiq-iteration 0.3.0 → 0.4.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +30 -0
- data/README.md +8 -0
- data/guides/best-practices.md +1 -1
- data/guides/custom-enumerator.md +1 -1
- data/guides/iteration-how-it-works.md +6 -0
- data/guides/throttling.md +1 -1
- data/lib/sidekiq_iteration/active_record_enumerator.rb +137 -77
- data/lib/sidekiq_iteration/csv_enumerator.rb +1 -1
- data/lib/sidekiq_iteration/enumerators.rb +3 -7
- data/lib/sidekiq_iteration/iteration.rb +10 -10
- data/lib/sidekiq_iteration/version.rb +1 -1
- data/lib/sidekiq_iteration.rb +10 -2
- metadata +3 -3
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: ea1fc3e6f5faff037ecfade45cc58bd2035385cafb888ef67ce253d12295aee8
|
4
|
+
data.tar.gz: a4bf097d4a1a8750f4d3e2ac9ce4f01191b5e7313635fed3d958b89647639c9a
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 415f27277011c3721853ae64c1c78c566a3bfe2c690ebaeb8e9e05bb10b4f3faac00cda0f6f95ef9fd980993695edc6621085833c7901bb09b14ff6027467622
|
7
|
+
data.tar.gz: 68568c8205c0370a3d1765e5bd6431655d1a3d5fd0ff07ee65fc6e8bf1dd0447fba1a62978c45fea3850029a9046f4cbe7768bce885f5bcb4d9c8e322a539ab6
|
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,35 @@
|
|
1
1
|
## master (unreleased)
|
2
2
|
|
3
|
+
## 0.4.0 (2024-05-10)
|
4
|
+
|
5
|
+
- Support ordering using multiple directions for ActiveRecord enumerators
|
6
|
+
|
7
|
+
```ruby
|
8
|
+
active_record_records_enumerator(..., columns: [:shop_id, :id], order: [:asc, :desc])
|
9
|
+
```
|
10
|
+
|
11
|
+
- Support iterating over ActiveRecord models with composite primary keys
|
12
|
+
|
13
|
+
- Use Arel to generate SQL in ActiveRecord enumerator
|
14
|
+
|
15
|
+
Previously, the enumerator coerced numeric ids to a string value (e.g.: `... AND id > '1'`),
|
16
|
+
which can cause problems on some DBMSes (like BigQuery).
|
17
|
+
|
18
|
+
- Enforce explicitly passed to ActiveRecord enumerators `:columns` value to include a primary key
|
19
|
+
|
20
|
+
Previously, the primary key column was added implicitly if it was not in the list.
|
21
|
+
|
22
|
+
```ruby
|
23
|
+
# before
|
24
|
+
active_record_records_enumerator(..., columns: [:updated_at])
|
25
|
+
|
26
|
+
# after
|
27
|
+
active_record_records_enumerator(..., columns: [:updated_at, :id])
|
28
|
+
```
|
29
|
+
|
30
|
+
- Accept single values as a `:columns` for ActiveRecord enumerators
|
31
|
+
- Add `around_iteration` hook
|
32
|
+
|
3
33
|
## 0.3.0 (2023-05-20)
|
4
34
|
|
5
35
|
- Allow a default retry backoff to be configured
|
data/README.md
CHANGED
@@ -4,6 +4,8 @@
|
|
4
4
|
|
5
5
|
Meet Iteration, an extension for [Sidekiq](https://github.com/mperham/sidekiq) that makes your long-running jobs interruptible and resumable, saving all progress that the job has made (aka checkpoint for jobs).
|
6
6
|
|
7
|
+
You may consider [`pluck_in_batches`](https://github.com/fatkodima/pluck_in_batches) gem to speedup iterating over large database tables.
|
8
|
+
|
7
9
|
## Background
|
8
10
|
|
9
11
|
Imagine the following job:
|
@@ -99,6 +101,12 @@ class NotifyUsersJob
|
|
99
101
|
# Will be called when the job starts iterating. Called only once, for the first time.
|
100
102
|
end
|
101
103
|
|
104
|
+
def around_iteration
|
105
|
+
# Will be called around each iteration.
|
106
|
+
# Can be useful for some metrics collection, performance tracking etc.
|
107
|
+
yield
|
108
|
+
end
|
109
|
+
|
102
110
|
def on_resume
|
103
111
|
# Called when the job resumes iterating.
|
104
112
|
end
|
data/guides/best-practices.md
CHANGED
@@ -2,7 +2,7 @@
|
|
2
2
|
|
3
3
|
## Considerations when writing jobs
|
4
4
|
|
5
|
-
* Duration of `#each_iteration`: processing a single element from the enumerator
|
5
|
+
* Duration of `#each_iteration`: processing a single element from the enumerator built in `#build_enumerator` should take less than 25 seconds, or the duration set as a timeout for Sidekiq. It allows the job to be safely interrupted and resumed.
|
6
6
|
* Idempotency of `#each_iteration`: it should be safe to run `#each_iteration` multiple times for the same element from the enumerator. Read more in [this Sidekiq best practice](https://github.com/mperham/sidekiq/wiki/Best-Practices#2-make-your-job-idempotent-and-transactional). It's important if the job errors and you run it again, because the same element that errored the job may be processed again. It especially matters in the situation described above, when the iteration duration exceeds the timeout: if the job is re-enqueued, multiple elements may be processed again.
|
7
7
|
|
8
8
|
## Batch iteration
|
data/guides/custom-enumerator.md
CHANGED
@@ -6,7 +6,7 @@ Before writing an enumerator, it is important to understand [how Iteration works
|
|
6
6
|
your enumerator will be used by it. An enumerator must `yield` two things in the following order as positional
|
7
7
|
arguments:
|
8
8
|
- An object to be processed in a job `each_iteration` method
|
9
|
-
- A cursor position, which Iteration will persist if `each_iteration` returns
|
9
|
+
- A cursor position, which Iteration will persist if `each_iteration` returns successfully and the job is forced to shut
|
10
10
|
down. It can be any data type your job backend can serialize and deserialize correctly.
|
11
11
|
|
12
12
|
A job that includes Iteration is first started with `nil` as the cursor. When resuming an interrupted job, Iteration
|
@@ -36,6 +36,12 @@ SELECT "users".* FROM "users" ORDER BY "users"."id" LIMIT 100
|
|
36
36
|
SELECT "users".* FROM "users" WHERE "users"."id" > 2 ORDER BY "products"."id" LIMIT 100
|
37
37
|
```
|
38
38
|
|
39
|
+
## Exceptions inside `each_iteration`
|
40
|
+
|
41
|
+
When an unrescued exception happens inside the `each_iteration` block, the job will stop and re-enqueue itself with the last successful cursor. This means that the iteration that failed will be retried with the same parameters and the cursor will only move if that iteration succeeds. This behaviour may be enough for intermittent errors, such as network connection failures, but if your execution is deterministic and you have an error, subsequent iterations will never run.
|
42
|
+
|
43
|
+
In other words, if you are trying to process 100 records but the job consistently fails on the 61st, only the first 60 will be processed and the job will try to process the 61st record until retries are exhausted.
|
44
|
+
|
39
45
|
## Signals
|
40
46
|
|
41
47
|
It's critical to know [UNIX signals](https://www.tutorialspoint.com/unix/unix-signals-traps.htm) in order to understand how interruption works. There are two main signals that Sidekiq use: `SIGTERM` and `SIGKILL`. `SIGTERM` is the graceful termination signal which means that the process should exit _soon_, not immediately. For Iteration, it means that we have time to wait for the last iteration to finish and to push job back to the queue with the last cursor position.
|
data/guides/throttling.md
CHANGED
@@ -25,7 +25,7 @@ class DeleteAccountsThrottledJob
|
|
25
25
|
end
|
26
26
|
```
|
27
27
|
|
28
|
-
Note that it
|
28
|
+
Note that it's up to you to define a throttling condition that makes sense for your app.
|
29
29
|
For example, `DatabaseStatus.healthy?` can check various MySQL metrics such as replication lag, DB threads, whether DB writes are available, etc.
|
30
30
|
|
31
31
|
Jobs can define multiple throttle conditions. Throttle conditions are inherited by descendants, and new conditions will be appended without impacting existing conditions.
|
@@ -10,42 +10,64 @@ module SidekiqIteration
|
|
10
10
|
raise ArgumentError, "relation must be an ActiveRecord::Relation"
|
11
11
|
end
|
12
12
|
|
13
|
-
unless order == :asc || order == :desc
|
14
|
-
raise ArgumentError, ":order must be :asc or :desc, got #{order.inspect}"
|
15
|
-
end
|
16
|
-
|
17
|
-
@primary_key = "#{relation.table_name}.#{relation.primary_key}"
|
18
|
-
@columns = Array(columns&.map(&:to_s) || @primary_key)
|
19
|
-
@primary_key_index = @columns.index(@primary_key) || @columns.index(relation.primary_key)
|
20
|
-
@pluck_columns = if @primary_key_index
|
21
|
-
@columns
|
22
|
-
else
|
23
|
-
@columns + [@primary_key]
|
24
|
-
end
|
25
|
-
@batch_size = batch_size
|
26
|
-
@order = order
|
27
|
-
@cursor = Array.wrap(cursor)
|
28
|
-
raise ArgumentError, "Must specify at least one column" if @columns.empty?
|
29
|
-
if relation.joins_values.present? && !@columns.all?(/\./)
|
30
|
-
raise ArgumentError, "You need to specify fully-qualified columns if you join a table"
|
31
|
-
end
|
32
|
-
|
33
13
|
if relation.arel.orders.present? || relation.arel.taken.present?
|
34
14
|
raise ArgumentError,
|
35
15
|
"The relation cannot use ORDER BY or LIMIT due to the way how iteration with a cursor is designed. " \
|
36
16
|
"You can use other ways to limit the number of rows, e.g. a WHERE condition on the primary key column."
|
37
17
|
end
|
38
18
|
|
39
|
-
|
19
|
+
@relation = relation
|
20
|
+
@primary_key = relation.primary_key
|
21
|
+
columns = Array(columns || @primary_key).map(&:to_s)
|
22
|
+
|
23
|
+
if (Array(order) - [:asc, :desc]).any?
|
24
|
+
raise ArgumentError, ":order must be :asc or :desc or an array consisting of :asc or :desc, got #{order.inspect}"
|
25
|
+
end
|
26
|
+
|
27
|
+
if order.is_a?(Array) && order.size != columns.size
|
28
|
+
raise ArgumentError, ":order must include a direction for each batching column"
|
29
|
+
end
|
30
|
+
|
31
|
+
@primary_key_index = primary_key_index(columns, relation)
|
32
|
+
if @primary_key_index.nil? || (composite_primary_key? && @primary_key_index.any?(nil))
|
33
|
+
raise ArgumentError, ":columns must include a primary key columns"
|
34
|
+
end
|
35
|
+
|
36
|
+
@batch_size = batch_size
|
37
|
+
@order = batch_order(columns, order)
|
38
|
+
@cursor = Array(cursor)
|
39
|
+
|
40
|
+
if @cursor.present? && @cursor.size != columns.size
|
41
|
+
raise ArgumentError, ":cursor must include values for all the columns from :columns"
|
42
|
+
end
|
43
|
+
|
44
|
+
if columns.any?(/\W/)
|
45
|
+
arel_columns = columns.map.with_index do |column, i|
|
46
|
+
arel_column(column).as("cursor_column_#{i + 1}")
|
47
|
+
end
|
48
|
+
@cursor_columns = arel_columns.map { |column| column.right.to_s }
|
49
|
+
|
50
|
+
relation =
|
51
|
+
if relation.select_values.empty?
|
52
|
+
relation.select(@relation.arel_table[Arel.star], arel_columns)
|
53
|
+
else
|
54
|
+
relation.select(arel_columns)
|
55
|
+
end
|
56
|
+
else
|
57
|
+
@cursor_columns = columns
|
58
|
+
end
|
59
|
+
|
60
|
+
@columns = columns
|
61
|
+
ordering = @columns.zip(@order).to_h
|
40
62
|
@base_relation = relation.reorder(ordering)
|
41
63
|
@iteration_count = 0
|
42
64
|
end
|
43
65
|
|
44
66
|
def records
|
45
67
|
Enumerator.new(-> { records_size }) do |yielder|
|
46
|
-
batches.each do |batch, _|
|
68
|
+
batches.each do |batch, _| # rubocop:disable Style/HashEachMethods
|
47
69
|
batch.each do |record|
|
48
|
-
|
70
|
+
increment_iteration
|
49
71
|
yielder.yield(record, cursor_value(record))
|
50
72
|
end
|
51
73
|
end
|
@@ -55,7 +77,7 @@ module SidekiqIteration
|
|
55
77
|
def batches
|
56
78
|
Enumerator.new(-> { records_size }) do |yielder|
|
57
79
|
while (batch = next_batch(load: true))
|
58
|
-
|
80
|
+
increment_iteration
|
59
81
|
yielder.yield(batch, cursor_value(batch.last))
|
60
82
|
end
|
61
83
|
end
|
@@ -64,13 +86,44 @@ module SidekiqIteration
|
|
64
86
|
def relations
|
65
87
|
Enumerator.new(-> { relations_size }) do |yielder|
|
66
88
|
while (batch = next_batch(load: false))
|
67
|
-
|
89
|
+
increment_iteration
|
68
90
|
yielder.yield(batch, unwrap_array(@cursor))
|
69
91
|
end
|
70
92
|
end
|
71
93
|
end
|
72
94
|
|
73
95
|
private
|
96
|
+
def primary_key_index(columns, relation)
|
97
|
+
indexes = Array(@primary_key).map do |pk_column|
|
98
|
+
columns.index do |column|
|
99
|
+
column == pk_column ||
|
100
|
+
(column.include?(relation.table_name) && column.include?(pk_column))
|
101
|
+
end
|
102
|
+
end
|
103
|
+
|
104
|
+
if composite_primary_key?
|
105
|
+
indexes
|
106
|
+
else
|
107
|
+
indexes.first
|
108
|
+
end
|
109
|
+
end
|
110
|
+
|
111
|
+
def batch_order(columns, order)
|
112
|
+
if order.is_a?(Array)
|
113
|
+
order
|
114
|
+
else
|
115
|
+
[order] * columns.size
|
116
|
+
end
|
117
|
+
end
|
118
|
+
|
119
|
+
def arel_column(column)
|
120
|
+
if column.include?(".")
|
121
|
+
Arel.sql(column)
|
122
|
+
else
|
123
|
+
@relation.arel_table[column]
|
124
|
+
end
|
125
|
+
end
|
126
|
+
|
74
127
|
def records_size
|
75
128
|
@base_relation.count(:all)
|
76
129
|
end
|
@@ -81,8 +134,8 @@ module SidekiqIteration
|
|
81
134
|
|
82
135
|
def next_batch(load:)
|
83
136
|
batch_relation = @base_relation.limit(@batch_size)
|
84
|
-
if
|
85
|
-
batch_relation = batch_relation
|
137
|
+
if @cursor.present?
|
138
|
+
batch_relation = apply_cursor(batch_relation)
|
86
139
|
end
|
87
140
|
|
88
141
|
records = nil
|
@@ -98,9 +151,7 @@ module SidekiqIteration
|
|
98
151
|
cursor = cursor_values.last
|
99
152
|
return unless cursor.present?
|
100
153
|
|
101
|
-
|
102
|
-
cursor.pop unless @primary_key_index
|
103
|
-
@cursor = Array.wrap(cursor)
|
154
|
+
@cursor = Array(cursor)
|
104
155
|
|
105
156
|
# Yields relations by selecting the primary keys of records in the batch.
|
106
157
|
# Post.where(published: nil) results in an enumerator of relations like:
|
@@ -111,80 +162,89 @@ module SidekiqIteration
|
|
111
162
|
end
|
112
163
|
|
113
164
|
def pluck_columns(batch)
|
114
|
-
|
115
|
-
|
116
|
-
@pluck_columns.map { |column| column.to_s.split(".").last }
|
117
|
-
else
|
118
|
-
@pluck_columns
|
119
|
-
end
|
120
|
-
|
121
|
-
if columns.size == 1 # only the primary key
|
122
|
-
column_values = batch.pluck(columns.first)
|
165
|
+
if @cursor_columns.size == 1 # only the primary key
|
166
|
+
column_values = batch.pluck(@cursor_columns.first)
|
123
167
|
return [column_values, column_values]
|
124
168
|
end
|
125
169
|
|
126
|
-
column_values = batch.pluck(
|
127
|
-
|
128
|
-
|
170
|
+
column_values = batch.pluck(*@cursor_columns)
|
171
|
+
primary_key_values =
|
172
|
+
if composite_primary_key?
|
173
|
+
column_values.map { |values| values.values_at(*@primary_key_index) }
|
174
|
+
else
|
175
|
+
column_values.map { |values| values[@primary_key_index] }
|
176
|
+
end
|
129
177
|
|
130
|
-
serialize_column_values
|
178
|
+
column_values = serialize_column_values(column_values)
|
131
179
|
[column_values, primary_key_values]
|
132
180
|
end
|
133
181
|
|
134
182
|
def cursor_value(record)
|
135
|
-
positions = @
|
136
|
-
|
137
|
-
column_value(record[attribute_name])
|
183
|
+
positions = @cursor_columns.map do |column|
|
184
|
+
column_value(record[column])
|
138
185
|
end
|
139
186
|
|
140
187
|
unwrap_array(positions)
|
141
188
|
end
|
142
189
|
|
143
|
-
|
144
|
-
|
190
|
+
# (x, y) >= (a, b) iff (x > a or (x = a and y >= b))
|
191
|
+
# (x, y) <= (a, b) iff (x < a or (x = a and y <= b))
|
192
|
+
def apply_cursor(relation)
|
193
|
+
arel_columns = @columns.map { |column| arel_column(column) }
|
194
|
+
cursor_positions = arel_columns.zip(@cursor, cursor_operators)
|
145
195
|
|
146
|
-
|
147
|
-
|
148
|
-
|
149
|
-
|
150
|
-
|
151
|
-
|
152
|
-
|
153
|
-
|
154
|
-
|
196
|
+
where_clause = nil
|
197
|
+
cursor_positions.reverse_each.with_index do |(arel_column, value, operator), index|
|
198
|
+
where_clause =
|
199
|
+
if index == 0
|
200
|
+
arel_column.public_send(operator, value)
|
201
|
+
else
|
202
|
+
arel_column.public_send(operator, value).or(
|
203
|
+
arel_column.eq(value).and(where_clause),
|
204
|
+
)
|
205
|
+
end
|
155
206
|
end
|
156
207
|
|
157
|
-
|
208
|
+
relation.where(where_clause)
|
158
209
|
end
|
159
210
|
|
160
|
-
|
161
|
-
|
162
|
-
|
163
|
-
column = @columns[index]
|
211
|
+
def serialize_column_values(column_values)
|
212
|
+
column_values.map { |values| values.map { |value| column_value(value) } }
|
213
|
+
end
|
164
214
|
|
165
|
-
|
166
|
-
|
167
|
-
|
215
|
+
def column_value(value)
|
216
|
+
if value.is_a?(Time)
|
217
|
+
value.strftime(SQL_DATETIME_WITH_NSEC)
|
168
218
|
else
|
169
|
-
|
170
|
-
|
171
|
-
|
219
|
+
value
|
220
|
+
end
|
221
|
+
end
|
222
|
+
|
223
|
+
def cursor_operators
|
224
|
+
# Start from the record pointed by cursor when just starting.
|
225
|
+
@columns.zip(@order).map do |column, order|
|
226
|
+
if column == @columns.last
|
227
|
+
if order == :asc
|
228
|
+
first_iteration? ? :gteq : :gt
|
229
|
+
else
|
230
|
+
first_iteration? ? :lteq : :lt
|
231
|
+
end
|
172
232
|
else
|
173
|
-
|
233
|
+
order == :asc ? :gt : :lt
|
174
234
|
end
|
175
235
|
end
|
176
236
|
end
|
177
237
|
|
178
|
-
def
|
179
|
-
|
238
|
+
def increment_iteration
|
239
|
+
@iteration_count += 1
|
180
240
|
end
|
181
241
|
|
182
|
-
def
|
183
|
-
|
184
|
-
|
185
|
-
|
186
|
-
|
187
|
-
|
242
|
+
def first_iteration?
|
243
|
+
@iteration_count == 0
|
244
|
+
end
|
245
|
+
|
246
|
+
def composite_primary_key?
|
247
|
+
@primary_key.is_a?(Array)
|
188
248
|
end
|
189
249
|
|
190
250
|
def unwrap_array(array)
|
@@ -36,7 +36,7 @@ module SidekiqIteration
|
|
36
36
|
# SidekiqIteration::CsvEnumerator.new(csv).rows(cursor: cursor)
|
37
37
|
#
|
38
38
|
def initialize(csv)
|
39
|
-
unless csv.instance_of?(CSV)
|
39
|
+
unless defined?(CSV) && csv.instance_of?(CSV)
|
40
40
|
raise ArgumentError, "CsvEnumerator.new takes CSV object"
|
41
41
|
end
|
42
42
|
|
@@ -17,10 +17,6 @@ module SidekiqIteration
|
|
17
17
|
def array_enumerator(array, cursor:)
|
18
18
|
raise ArgumentError, "array must be an Array" unless array.is_a?(Array)
|
19
19
|
|
20
|
-
if defined?(ActiveRecord) && array.any?(ActiveRecord::Base)
|
21
|
-
raise ArgumentError, "array cannot contain ActiveRecord objects"
|
22
|
-
end
|
23
|
-
|
24
20
|
array.each_with_index.drop(cursor || 0).to_enum { array.size }
|
25
21
|
end
|
26
22
|
|
@@ -28,10 +24,10 @@ module SidekiqIteration
|
|
28
24
|
#
|
29
25
|
# @param scope [ActiveRecord::Relation] scope to iterate
|
30
26
|
# @param cursor [Object] offset to start iteration from, usually an id
|
31
|
-
# @option options :columns [Array<String, Symbol
|
27
|
+
# @option options :columns [Array<String, Symbol>, String, Symbol] used to build the actual query for iteration,
|
32
28
|
# defaults to primary key
|
33
29
|
# @option options :batch_size [Integer] (100) size of the batch
|
34
|
-
# @option options :order [:asc, :desc] (:asc) specifies iteration order
|
30
|
+
# @option options :order [:asc, :desc, Array<:asc, :desc>] (:asc) specifies iteration order
|
35
31
|
#
|
36
32
|
# +columns:+ argument is used to build the actual query for iteration. +columns+: defaults to primary key:
|
37
33
|
#
|
@@ -59,7 +55,7 @@ module SidekiqIteration
|
|
59
55
|
# As a result of this query pattern, if the values in these columns change for the records in scope during
|
60
56
|
# iteration, they may be skipped or yielded multiple times depending on the nature of the update and the
|
61
57
|
# cursor's value. If the value gets updated to a greater value than the cursor's value, it will get yielded
|
62
|
-
# again. Similarly, if the value gets updated to a lesser value than the
|
58
|
+
# again. Similarly, if the value gets updated to a lesser value than the cursor's value, it will get skipped.
|
63
59
|
#
|
64
60
|
# @example
|
65
61
|
# def build_enumerator(cursor:)
|
@@ -20,8 +20,7 @@ module SidekiqIteration
|
|
20
20
|
end
|
21
21
|
|
22
22
|
throttle_on(backoff: SidekiqIteration.default_retry_backoff) do
|
23
|
-
|
24
|
-
Sidekiq::CLI.instance.launcher.stopping?
|
23
|
+
SidekiqIteration.stopping
|
25
24
|
end
|
26
25
|
end
|
27
26
|
|
@@ -88,6 +87,12 @@ module SidekiqIteration
|
|
88
87
|
def on_start
|
89
88
|
end
|
90
89
|
|
90
|
+
# A hook to override that will be called around each iteration.
|
91
|
+
# Can be useful for some metrics collection, performance tracking etc.
|
92
|
+
def around_iteration
|
93
|
+
yield
|
94
|
+
end
|
95
|
+
|
91
96
|
# A hook to override that will be called when the job resumes iterating.
|
92
97
|
def on_resume
|
93
98
|
end
|
@@ -178,7 +183,9 @@ module SidekiqIteration
|
|
178
183
|
|
179
184
|
enumerator.each do |object_from_enumerator, index|
|
180
185
|
found_record = true
|
181
|
-
|
186
|
+
around_iteration do
|
187
|
+
each_iteration(object_from_enumerator, *arguments)
|
188
|
+
end
|
182
189
|
@cursor_position = index
|
183
190
|
@current_run_iterations += 1
|
184
191
|
|
@@ -258,13 +265,6 @@ module SidekiqIteration
|
|
258
265
|
true
|
259
266
|
when false, :skip_complete_callback
|
260
267
|
false
|
261
|
-
when Array # can be used to return early from the enumerator
|
262
|
-
reason, backoff = completed
|
263
|
-
raise "Unknown reason: #{reason}" unless reason == :retry
|
264
|
-
|
265
|
-
@job_iteration_retry_backoff = backoff
|
266
|
-
@needs_reenqueue = true
|
267
|
-
false
|
268
268
|
else
|
269
269
|
raise "Unexpected thrown value: #{completed.inspect}"
|
270
270
|
end
|
data/lib/sidekiq_iteration.rb
CHANGED
@@ -1,6 +1,8 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
require "sidekiq"
|
4
|
+
require_relative "sidekiq_iteration/iteration"
|
5
|
+
require_relative "sidekiq_iteration/job_retry_patch"
|
4
6
|
require_relative "sidekiq_iteration/version"
|
5
7
|
|
6
8
|
module SidekiqIteration
|
@@ -44,8 +46,14 @@ module SidekiqIteration
|
|
44
46
|
def logger
|
45
47
|
@logger ||= Sidekiq.logger
|
46
48
|
end
|
49
|
+
|
50
|
+
# @private
|
51
|
+
attr_accessor :stopping
|
47
52
|
end
|
48
53
|
end
|
49
54
|
|
50
|
-
|
51
|
-
|
55
|
+
Sidekiq.configure_server do |config|
|
56
|
+
config.on(:quiet) do
|
57
|
+
SidekiqIteration.stopping = true
|
58
|
+
end
|
59
|
+
end
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: sidekiq-iteration
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.4.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- fatkodima
|
@@ -9,7 +9,7 @@ authors:
|
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date:
|
12
|
+
date: 2024-05-10 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: sidekiq
|
@@ -72,7 +72,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
72
72
|
- !ruby/object:Gem::Version
|
73
73
|
version: '0'
|
74
74
|
requirements: []
|
75
|
-
rubygems_version: 3.4.
|
75
|
+
rubygems_version: 3.4.19
|
76
76
|
signing_key:
|
77
77
|
specification_version: 4
|
78
78
|
summary: Makes your long-running sidekiq jobs interruptible and resumable.
|