cloudtasker 0.12.rc4 → 0.12.rc9
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.rubocop.yml +2 -2
- data/CHANGELOG.md +11 -4
- data/README.md +60 -0
- data/app/controllers/cloudtasker/worker_controller.rb +1 -1
- data/docs/BATCH_JOBS.md +24 -3
- data/lib/cloudtasker/backend/memory_task.rb +1 -1
- data/lib/cloudtasker/backend/redis_task.rb +17 -6
- data/lib/cloudtasker/batch/job.rb +54 -22
- data/lib/cloudtasker/cloud_task.rb +3 -2
- data/lib/cloudtasker/config.rb +16 -1
- data/lib/cloudtasker/redis_client.rb +6 -2
- data/lib/cloudtasker/version.rb +1 -1
- data/lib/cloudtasker/worker.rb +33 -6
- data/lib/cloudtasker/worker_handler.rb +5 -13
- metadata +2 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 50cd6021ad7511d7b5b385a6086eb315d9381a4f43909a6e1e9a0defbc679ea4
|
4
|
+
data.tar.gz: dc16cc1330b14b46a5d6ade9f9c171981fd3aef5b3db67dd474e29548acc177f
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 30feac9331b9e8113e71af6c44a20d2f47fcbd41652a0d345ab8808d272bd14271ebb4a98711e42b238f6202fa8023a37c543e0f4ec7d9f9f7a46cc2732d96aa
|
7
|
+
data.tar.gz: 9b70a3c0733b2c510fafe48232c65bb79f5ad4da6b90fbcff86c0736fee590fab5f89846df9c41855dc7084460425660ee42799975975775b2239b9b8997171a
|
data/.rubocop.yml
CHANGED
data/CHANGELOG.md
CHANGED
@@ -1,15 +1,22 @@
|
|
1
1
|
# Changelog
|
2
2
|
|
3
|
-
## Latest RC [v0.12.
|
3
|
+
## Latest RC [v0.12.rc8](https://github.com/keypup-io/cloudtasker/tree/v0.12.rc8) (2021-04-06)
|
4
4
|
|
5
|
-
[Full Changelog](https://github.com/keypup-io/cloudtasker/compare/v0.11.0...v0.12.
|
5
|
+
[Full Changelog](https://github.com/keypup-io/cloudtasker/compare/v0.11.0...v0.12.rc8)
|
6
6
|
|
7
7
|
**Improvements:**
|
8
8
|
- ActiveJob: do not double log errors (ActiveJob has its own error logging)
|
9
|
+
- Batch callbacks: Retry jobs when completion callback fails
|
10
|
+
- Batch state: use native Redis hashes to store batch state instead of a serialized hash in a string key
|
11
|
+
- Batch progress: restrict calculation to direct children by default. Allow depth to be specified. Calculating progress using all tree jobs created significant delays on large batches.
|
12
|
+
- Batch redis usage: cleanup batches as they get completed or become dead to avoid excessive redis usage with large batches.
|
13
|
+
- Configuration: allow configuration of Cloud Tasks `dispatch deadline` at global and worker level
|
14
|
+
- Cron jobs: Use Redis Sets instead of key pattern matching for resource listing
|
9
15
|
- Error logging: Use worker logger so as to include context (job args etc.)
|
10
16
|
- Error logging: Do not log exception and stack trace separately, combine them instead.
|
11
|
-
-
|
12
|
-
-
|
17
|
+
- Local server: Use Redis Sets instead of key pattern matching for resource listing
|
18
|
+
- Worker: raise DeadWorkerError instead of MissingWorkerArgumentsError when arguments are missing. This is more consistent with what middlewares expect.
|
19
|
+
- Worker redis usage: delete redis payload storage once the job is successful or dead instead of expiring the key.
|
13
20
|
|
14
21
|
**Fixed bugs:**
|
15
22
|
- Retries: Enforce job retry limit on job processing. There was an edge case where jobs could be retried indefinitely on batch callback errors.
|
data/README.md
CHANGED
@@ -37,6 +37,7 @@ A local processing server is also available for development. This local server p
|
|
37
37
|
1. [HTTP Error codes](#http-error-codes)
|
38
38
|
2. [Error callbacks](#error-callbacks)
|
39
39
|
3. [Max retries](#max-retries)
|
40
|
+
4. [Dispatch deadline](#dispatch-deadline)
|
40
41
|
10. [Testing](#testing)
|
41
42
|
1. [Test helper setup](#test-helper-setup)
|
42
43
|
2. [In-memory queues](#in-memory-queues)
|
@@ -351,6 +352,23 @@ Cloudtasker.configure do |config|
|
|
351
352
|
#
|
352
353
|
# Store all job payloads in Redis exceeding 50 KB:
|
353
354
|
# config.store_payloads_in_redis = 50
|
355
|
+
|
356
|
+
#
|
357
|
+
# Specify the dispatch deadline for jobs in Cloud Tasks, in seconds.
|
358
|
+
# Jobs taking longer will be retried by Cloud Tasks, even if they eventually
|
359
|
+
# complete on the server side.
|
360
|
+
#
|
361
|
+
# Note that this option is applied when jobs are enqueued job. Changing this value
|
362
|
+
# will not impact already enqueued jobs.
|
363
|
+
#
|
364
|
+
# This option can also be configured on a per worker basis via
|
365
|
+
# the cloudtasker_options directive.
|
366
|
+
#
|
367
|
+
# Supported since: v0.12.rc8
|
368
|
+
#
|
369
|
+
# Default: 600 seconds (10 minutes)
|
370
|
+
#
|
371
|
+
# config.dispatch_deadline = 600
|
354
372
|
end
|
355
373
|
```
|
356
374
|
|
@@ -721,6 +739,48 @@ class SomeErrorWorker
|
|
721
739
|
end
|
722
740
|
```
|
723
741
|
|
742
|
+
### Dispatch deadline
|
743
|
+
**Supported since**: `0.12.rc8`
|
744
|
+
|
745
|
+
By default Cloud Tasks will automatically timeout your jobs after 10 minutes, independently of your server HTTP timeout configuration.
|
746
|
+
|
747
|
+
You can modify the dispatch deadline for jobs at a global level or on a per job basis.
|
748
|
+
|
749
|
+
E.g. Set the default dispatch deadline to 20 minutes.
|
750
|
+
```ruby
|
751
|
+
# config/initializers/cloudtasker.rb
|
752
|
+
|
753
|
+
Cloudtasker.configure do |config|
|
754
|
+
#
|
755
|
+
# Specify the dispatch deadline for jobs in Cloud Tasks, in seconds.
|
756
|
+
# Jobs taking longer will be retried by Cloud Tasks, even if they eventually
|
757
|
+
# complete on the server side.
|
758
|
+
#
|
759
|
+
# Note that this option is applied when jobs are enqueued job. Changing this value
|
760
|
+
# will not impact already enqueued jobs.
|
761
|
+
#
|
762
|
+
# Default: 600 (10 minutes)
|
763
|
+
#
|
764
|
+
config.dispatch_deadline = 20 * 60 # 20 minutes
|
765
|
+
end
|
766
|
+
```
|
767
|
+
|
768
|
+
E.g. Set a dispatch deadline of 5 minutes on a specific worker
|
769
|
+
```ruby
|
770
|
+
# app/workers/some_error_worker.rb
|
771
|
+
|
772
|
+
class SomeFasterWorker
|
773
|
+
include Cloudtasker::Worker
|
774
|
+
|
775
|
+
# This will override the global setting
|
776
|
+
cloudtasker_options dispatch_deadline: 5 * 60
|
777
|
+
|
778
|
+
def perform
|
779
|
+
# ... do things ...
|
780
|
+
end
|
781
|
+
end
|
782
|
+
```
|
783
|
+
|
724
784
|
## Testing
|
725
785
|
Cloudtasker provides several options to test your workers.
|
726
786
|
|
@@ -19,7 +19,7 @@ module Cloudtasker
|
|
19
19
|
# Process payload
|
20
20
|
WorkerHandler.execute_from_payload!(payload)
|
21
21
|
head :no_content
|
22
|
-
rescue DeadWorkerError
|
22
|
+
rescue DeadWorkerError
|
23
23
|
# 205: job will NOT be retried
|
24
24
|
head :reset_content
|
25
25
|
rescue InvalidWorkerError
|
data/docs/BATCH_JOBS.md
CHANGED
@@ -84,8 +84,29 @@ You can access progression statistics in callback using `batch.progress`. See th
|
|
84
84
|
E.g.
|
85
85
|
```ruby
|
86
86
|
def on_batch_node_complete(_child_job)
|
87
|
-
|
88
|
-
logger.info("
|
89
|
-
logger.info("
|
87
|
+
progress = batch.progress
|
88
|
+
logger.info("Total: #{progress.total}")
|
89
|
+
logger.info("Completed: #{progress.completed}")
|
90
|
+
logger.info("Progress: #{progress.percent.to_i}%")
|
91
|
+
end
|
92
|
+
```
|
93
|
+
|
94
|
+
**Since:** `v0.12.rc5`
|
95
|
+
By default the `progress` method only considers the direct child jobs to evaluate the batch progress. You can pass `depth: somenumber` to the `progress` method to calculate the actual batch progress in a more granular way. Be careful however that this method recursively calculates progress on the sub-batches and is therefore expensive.
|
96
|
+
|
97
|
+
E.g.
|
98
|
+
```ruby
|
99
|
+
def on_batch_node_complete(_child_job)
|
100
|
+
# Considers the children for batch progress calculation
|
101
|
+
progress_0 = batch.progress # same as batch.progress(depth: 0)
|
102
|
+
|
103
|
+
# Considers the children and grand-children for batch progress calculation
|
104
|
+
progress_1 = batch.progress(depth: 1)
|
105
|
+
|
106
|
+
# Considers the children, grand-children and grand-grand-children for batch progress calculation
|
107
|
+
progress_2 = batch.progress(depth: 3)
|
108
|
+
|
109
|
+
logger.info("Progress: #{progress_1.percent.to_i}%")
|
110
|
+
logger.info("Progress: #{progress_2.percent.to_i}%")
|
90
111
|
end
|
91
112
|
```
|
@@ -113,7 +113,7 @@ module Cloudtasker
|
|
113
113
|
# @param [Hash] http_request The HTTP request content.
|
114
114
|
# @param [Integer] schedule_time When to run the task (Unix timestamp)
|
115
115
|
#
|
116
|
-
def initialize(id:, http_request:, schedule_time: nil, queue: nil, job_retries: 0)
|
116
|
+
def initialize(id:, http_request:, schedule_time: nil, queue: nil, job_retries: 0, **_xargs)
|
117
117
|
@id = id
|
118
118
|
@http_request = http_request
|
119
119
|
@schedule_time = Time.at(schedule_time || 0)
|
@@ -7,7 +7,7 @@ module Cloudtasker
|
|
7
7
|
module Backend
|
8
8
|
# Manage local tasks pushed to Redis
|
9
9
|
class RedisTask
|
10
|
-
attr_reader :id, :http_request, :schedule_time, :retries, :queue
|
10
|
+
attr_reader :id, :http_request, :schedule_time, :retries, :queue, :dispatch_deadline
|
11
11
|
|
12
12
|
RETRY_INTERVAL = 20 # seconds
|
13
13
|
|
@@ -123,13 +123,15 @@ module Cloudtasker
|
|
123
123
|
# @param [Hash] http_request The HTTP request content.
|
124
124
|
# @param [Integer] schedule_time When to run the task (Unix timestamp)
|
125
125
|
# @param [Integer] retries The number of times the job failed.
|
126
|
+
# @param [Integer] dispatch_deadline The dispatch_deadline in seconds.
|
126
127
|
#
|
127
|
-
def initialize(id:, http_request:, schedule_time: nil, retries: 0, queue: nil)
|
128
|
+
def initialize(id:, http_request:, schedule_time: nil, retries: 0, queue: nil, dispatch_deadline: nil)
|
128
129
|
@id = id
|
129
130
|
@http_request = http_request
|
130
131
|
@schedule_time = Time.at(schedule_time || 0)
|
131
132
|
@retries = retries || 0
|
132
|
-
@queue = queue ||
|
133
|
+
@queue = queue || Config::DEFAULT_JOB_QUEUE
|
134
|
+
@dispatch_deadline = dispatch_deadline || Config::DEFAULT_DISPATCH_DEADLINE
|
133
135
|
end
|
134
136
|
|
135
137
|
#
|
@@ -152,7 +154,8 @@ module Cloudtasker
|
|
152
154
|
http_request: http_request,
|
153
155
|
schedule_time: schedule_time.to_i,
|
154
156
|
retries: retries,
|
155
|
-
queue: queue
|
157
|
+
queue: queue,
|
158
|
+
dispatch_deadline: dispatch_deadline
|
156
159
|
}
|
157
160
|
end
|
158
161
|
|
@@ -176,7 +179,8 @@ module Cloudtasker
|
|
176
179
|
retries: is_error ? retries + 1 : retries,
|
177
180
|
http_request: http_request,
|
178
181
|
schedule_time: (Time.now + interval).to_i,
|
179
|
-
queue: queue
|
182
|
+
queue: queue,
|
183
|
+
dispatch_deadline: dispatch_deadline
|
180
184
|
)
|
181
185
|
redis.sadd(self.class.key, id)
|
182
186
|
end
|
@@ -207,6 +211,13 @@ module Cloudtasker
|
|
207
211
|
end
|
208
212
|
|
209
213
|
resp
|
214
|
+
rescue Net::ReadTimeout
|
215
|
+
retry_later(RETRY_INTERVAL)
|
216
|
+
Cloudtasker.logger.info(
|
217
|
+
format_log_message(
|
218
|
+
"Task deadline exceeded (#{dispatch_deadline}s) - Retry in #{RETRY_INTERVAL} seconds..."
|
219
|
+
)
|
220
|
+
)
|
210
221
|
end
|
211
222
|
|
212
223
|
#
|
@@ -242,7 +253,7 @@ module Cloudtasker
|
|
242
253
|
@http_client ||=
|
243
254
|
begin
|
244
255
|
uri = URI(http_request[:url])
|
245
|
-
Net::HTTP.new(uri.host, uri.port).tap { |e| e.read_timeout =
|
256
|
+
Net::HTTP.new(uri.host, uri.port).tap { |e| e.read_timeout = dispatch_deadline }
|
246
257
|
end
|
247
258
|
end
|
248
259
|
|
@@ -17,6 +17,10 @@ module Cloudtasker
|
|
17
17
|
# because the jobs will be either retried or dropped
|
18
18
|
IGNORED_ERRORED_CALLBACKS = %i[on_child_error on_child_dead].freeze
|
19
19
|
|
20
|
+
# The maximum number of seconds to wait for a batch state lock
|
21
|
+
# to be acquired.
|
22
|
+
BATCH_MAX_LOCK_WAIT = 60
|
23
|
+
|
20
24
|
#
|
21
25
|
# Return the cloudtasker redis client
|
22
26
|
#
|
@@ -176,7 +180,9 @@ module Cloudtasker
|
|
176
180
|
# @return [Hash] The state of each child worker.
|
177
181
|
#
|
178
182
|
def batch_state
|
179
|
-
|
183
|
+
migrate_batch_state_to_redis_hash
|
184
|
+
|
185
|
+
redis.hgetall(batch_state_gid)
|
180
186
|
end
|
181
187
|
|
182
188
|
#
|
@@ -208,6 +214,24 @@ module Cloudtasker
|
|
208
214
|
)
|
209
215
|
end
|
210
216
|
|
217
|
+
#
|
218
|
+
# This method migrates the batch state to be a Redis hash instead
|
219
|
+
# of a hash stored in a string key.
|
220
|
+
#
|
221
|
+
def migrate_batch_state_to_redis_hash
|
222
|
+
return unless redis.type(batch_state_gid) == 'string'
|
223
|
+
|
224
|
+
# Migrate batch state to Redis hash if it is still using a legacy string key
|
225
|
+
# We acquire a lock then check again
|
226
|
+
redis.with_lock(batch_state_gid, max_wait: BATCH_MAX_LOCK_WAIT) do
|
227
|
+
if redis.type(batch_state_gid) == 'string'
|
228
|
+
state = redis.fetch(batch_state_gid)
|
229
|
+
redis.del(batch_state_gid)
|
230
|
+
redis.hset(batch_state_gid, state) if state.any?
|
231
|
+
end
|
232
|
+
end
|
233
|
+
end
|
234
|
+
|
211
235
|
#
|
212
236
|
# Save the batch.
|
213
237
|
#
|
@@ -218,8 +242,11 @@ module Cloudtasker
|
|
218
242
|
# complete (success or failure).
|
219
243
|
redis.write(batch_gid, worker.to_h)
|
220
244
|
|
245
|
+
# Stop there if no jobs to save
|
246
|
+
return if jobs.empty?
|
247
|
+
|
221
248
|
# Save list of child workers
|
222
|
-
redis.
|
249
|
+
redis.hset(batch_state_gid, jobs.map { |e| [e.job_id, 'scheduled'] }.to_h)
|
223
250
|
end
|
224
251
|
|
225
252
|
#
|
@@ -228,28 +255,27 @@ module Cloudtasker
|
|
228
255
|
# @param [String] job_id The batch id.
|
229
256
|
# @param [String] status The status of the sub-batch.
|
230
257
|
#
|
231
|
-
# @return [<Type>] <description>
|
232
|
-
#
|
233
258
|
def update_state(batch_id, status)
|
234
|
-
|
235
|
-
|
236
|
-
|
237
|
-
|
259
|
+
migrate_batch_state_to_redis_hash
|
260
|
+
|
261
|
+
# Update the batch state batch_id entry with the new status
|
262
|
+
redis.with_lock("#{batch_state_gid}/#{batch_id}", max_wait: BATCH_MAX_LOCK_WAIT) do
|
263
|
+
redis.hset(batch_state_gid, batch_id, status) if redis.hexists(batch_state_gid, batch_id)
|
238
264
|
end
|
239
265
|
end
|
240
266
|
|
241
267
|
#
|
242
268
|
# Return true if all the child workers have completed.
|
243
269
|
#
|
244
|
-
# @return [
|
270
|
+
# @return [Boolean] True if the batch is complete.
|
245
271
|
#
|
246
272
|
def complete?
|
247
|
-
|
248
|
-
state = redis.fetch(batch_state_gid)
|
249
|
-
return true unless state
|
273
|
+
migrate_batch_state_to_redis_hash
|
250
274
|
|
275
|
+
# Check that all child jobs have completed
|
276
|
+
redis.with_lock(batch_state_gid, max_wait: BATCH_MAX_LOCK_WAIT) do
|
251
277
|
# Check that all children are complete
|
252
|
-
|
278
|
+
redis.hvals(batch_state_gid).all? { |e| COMPLETION_STATUSES.include?(e) }
|
253
279
|
end
|
254
280
|
end
|
255
281
|
|
@@ -285,8 +311,8 @@ module Cloudtasker
|
|
285
311
|
# Propagate event
|
286
312
|
parent_batch&.on_child_complete(self, status)
|
287
313
|
|
288
|
-
# The batch tree is complete. Cleanup the tree.
|
289
|
-
cleanup
|
314
|
+
# The batch tree is complete. Cleanup the downstream tree.
|
315
|
+
cleanup
|
290
316
|
end
|
291
317
|
|
292
318
|
#
|
@@ -331,11 +357,10 @@ module Cloudtasker
|
|
331
357
|
# Remove all batch and sub-batch keys from Redis.
|
332
358
|
#
|
333
359
|
def cleanup
|
334
|
-
|
335
|
-
state = batch_state
|
360
|
+
migrate_batch_state_to_redis_hash
|
336
361
|
|
337
362
|
# Delete child batches recursively
|
338
|
-
|
363
|
+
redis.hkeys(batch_state_gid).each { |id| self.class.find(id)&.cleanup }
|
339
364
|
|
340
365
|
# Delete batch redis entries
|
341
366
|
redis.del(batch_gid)
|
@@ -347,13 +372,20 @@ module Cloudtasker
|
|
347
372
|
#
|
348
373
|
# @return [Cloudtasker::Batch::BatchProgress] The batch progress.
|
349
374
|
#
|
350
|
-
def progress
|
375
|
+
def progress(depth: 0)
|
376
|
+
depth = depth.to_i
|
377
|
+
|
351
378
|
# Capture batch state
|
352
379
|
state = batch_state
|
353
380
|
|
354
|
-
#
|
381
|
+
# Return immediately if we do not need to go down the tree
|
382
|
+
return BatchProgress.new(state) if depth <= 0
|
383
|
+
|
384
|
+
# Sum batch progress of current batch and sub-batches up to the specified
|
385
|
+
# depth
|
355
386
|
state.to_h.reduce(BatchProgress.new(state)) do |memo, (child_id, child_status)|
|
356
|
-
memo + (self.class.find(child_id)&.progress
|
387
|
+
memo + (self.class.find(child_id)&.progress(depth: depth - 1) ||
|
388
|
+
BatchProgress.new(child_id => child_status))
|
357
389
|
end
|
358
390
|
end
|
359
391
|
|
@@ -395,7 +427,7 @@ module Cloudtasker
|
|
395
427
|
# Perform job
|
396
428
|
yield
|
397
429
|
|
398
|
-
# Save batch (if child
|
430
|
+
# Save batch (if child workers have been enqueued)
|
399
431
|
setup
|
400
432
|
|
401
433
|
# Complete batch
|
@@ -3,7 +3,7 @@
|
|
3
3
|
module Cloudtasker
|
4
4
|
# An interface class to manage tasks on the backend (Cloud Task or Redis)
|
5
5
|
class CloudTask
|
6
|
-
attr_accessor :id, :http_request, :schedule_time, :retries, :queue
|
6
|
+
attr_accessor :id, :http_request, :schedule_time, :retries, :queue, :dispatch_deadline
|
7
7
|
|
8
8
|
#
|
9
9
|
# The backend to use for cloud tasks.
|
@@ -73,12 +73,13 @@ module Cloudtasker
|
|
73
73
|
# @param [Integer] retries The number of times the job failed.
|
74
74
|
# @param [String] queue The queue the task is in.
|
75
75
|
#
|
76
|
-
def initialize(id:, http_request:, schedule_time: nil, retries: 0, queue: nil)
|
76
|
+
def initialize(id:, http_request:, schedule_time: nil, retries: 0, queue: nil, dispatch_deadline: nil)
|
77
77
|
@id = id
|
78
78
|
@http_request = http_request
|
79
79
|
@schedule_time = schedule_time
|
80
80
|
@retries = retries || 0
|
81
81
|
@queue = queue
|
82
|
+
@dispatch_deadline = dispatch_deadline
|
82
83
|
end
|
83
84
|
|
84
85
|
#
|
data/lib/cloudtasker/config.rb
CHANGED
@@ -7,7 +7,7 @@ module Cloudtasker
|
|
7
7
|
class Config
|
8
8
|
attr_accessor :redis, :store_payloads_in_redis
|
9
9
|
attr_writer :secret, :gcp_location_id, :gcp_project_id,
|
10
|
-
:gcp_queue_prefix, :processor_path, :logger, :mode, :max_retries
|
10
|
+
:gcp_queue_prefix, :processor_path, :logger, :mode, :max_retries, :dispatch_deadline
|
11
11
|
|
12
12
|
# Max Cloud Task size in bytes
|
13
13
|
MAX_TASK_SIZE = 100 * 1024 # 100 KB
|
@@ -46,6 +46,11 @@ module Cloudtasker
|
|
46
46
|
DEFAULT_QUEUE_CONCURRENCY = 10
|
47
47
|
DEFAULT_QUEUE_RETRIES = -1 # unlimited
|
48
48
|
|
49
|
+
# Job timeout configuration for Cloud Tasks
|
50
|
+
DEFAULT_DISPATCH_DEADLINE = 10 * 60 # 10 minutes
|
51
|
+
MIN_DISPATCH_DEADLINE = 15 # seconds
|
52
|
+
MAX_DISPATCH_DEADLINE = 30 * 60 # 30 minutes
|
53
|
+
|
49
54
|
# The number of times jobs will be attempted before declaring them dead.
|
50
55
|
#
|
51
56
|
# With the default retry configuration (maxDoublings = 16 and minBackoff = 0.100s)
|
@@ -207,6 +212,16 @@ module Cloudtasker
|
|
207
212
|
@gcp_location_id || DEFAULT_LOCATION_ID
|
208
213
|
end
|
209
214
|
|
215
|
+
#
|
216
|
+
# Return the Dispatch deadline duration. Cloud Tasks will timeout the job after
|
217
|
+
# this duration is elapsed.
|
218
|
+
#
|
219
|
+
# @return [Integer] The value in seconds.
|
220
|
+
#
|
221
|
+
def dispatch_deadline
|
222
|
+
@dispatch_deadline || DEFAULT_DISPATCH_DEADLINE
|
223
|
+
end
|
224
|
+
|
210
225
|
#
|
211
226
|
# Return the secret to use to sign the verification tokens
|
212
227
|
# attached to tasks.
|
@@ -75,14 +75,18 @@ module Cloudtasker
|
|
75
75
|
# end
|
76
76
|
#
|
77
77
|
# @param [String] cache_key The cache key to access.
|
78
|
+
# @param [Integer] max_wait The number of seconds after which the lock will be cleared anyway.
|
78
79
|
#
|
79
|
-
def with_lock(cache_key)
|
80
|
+
def with_lock(cache_key, max_wait: nil)
|
80
81
|
return nil unless cache_key
|
81
82
|
|
83
|
+
# Set max wait
|
84
|
+
max_wait = (max_wait || LOCK_DURATION).to_i
|
85
|
+
|
82
86
|
# Wait to acquire lock
|
83
87
|
lock_key = [LOCK_KEY_PREFIX, cache_key].join('/')
|
84
88
|
client.with do |conn|
|
85
|
-
sleep(LOCK_WAIT_DURATION) until conn.set(lock_key, true, nx: true, ex:
|
89
|
+
sleep(LOCK_WAIT_DURATION) until conn.set(lock_key, true, nx: true, ex: max_wait)
|
86
90
|
end
|
87
91
|
|
88
92
|
# yield content
|
data/lib/cloudtasker/version.rb
CHANGED
data/lib/cloudtasker/worker.rb
CHANGED
@@ -167,6 +167,22 @@ module Cloudtasker
|
|
167
167
|
(@job_queue ||= self.class.cloudtasker_options_hash[:queue] || Config::DEFAULT_JOB_QUEUE).to_s
|
168
168
|
end
|
169
169
|
|
170
|
+
#
|
171
|
+
# Return the Dispatch deadline duration. Cloud Tasks will timeout the job after
|
172
|
+
# this duration is elapsed.
|
173
|
+
#
|
174
|
+
# @return [Integer] The value in seconds.
|
175
|
+
#
|
176
|
+
def dispatch_deadline
|
177
|
+
@dispatch_deadline ||= [
|
178
|
+
[
|
179
|
+
Config::MIN_DISPATCH_DEADLINE,
|
180
|
+
(self.class.cloudtasker_options_hash[:dispatch_deadline] || Cloudtasker.config.dispatch_deadline).to_i
|
181
|
+
].max,
|
182
|
+
Config::MAX_DISPATCH_DEADLINE
|
183
|
+
].min
|
184
|
+
end
|
185
|
+
|
170
186
|
#
|
171
187
|
# Return the Cloudtasker logger instance.
|
172
188
|
#
|
@@ -332,6 +348,22 @@ module Cloudtasker
|
|
332
348
|
job_retries > job_max_retries
|
333
349
|
end
|
334
350
|
|
351
|
+
#
|
352
|
+
# Return true if the job arguments are missing.
|
353
|
+
#
|
354
|
+
# This may happen if a job
|
355
|
+
# was successfully run but retried due to Cloud Task dispatch deadline
|
356
|
+
# exceeded. If the arguments were stored in Redis then they may have
|
357
|
+
# been flushed already after the successful completion.
|
358
|
+
#
|
359
|
+
# If job arguments are missing then the job will simply be declared dead.
|
360
|
+
#
|
361
|
+
# @return [Boolean] True if the arguments are missing.
|
362
|
+
#
|
363
|
+
def arguments_missing?
|
364
|
+
job_args.empty? && [0, -1].exclude?(method(:perform).arity)
|
365
|
+
end
|
366
|
+
|
335
367
|
#
|
336
368
|
# Return the time taken (in seconds) to perform the job. This duration
|
337
369
|
# includes the middlewares and the actual perform method.
|
@@ -384,14 +416,9 @@ module Cloudtasker
|
|
384
416
|
Cloudtasker.config.server_middleware.invoke(self) do
|
385
417
|
# Immediately abort the job if it is already dead
|
386
418
|
flag_as_dead if job_dead?
|
419
|
+
flag_as_dead(MissingWorkerArgumentsError.new('worker arguments are missing')) if arguments_missing?
|
387
420
|
|
388
421
|
begin
|
389
|
-
# Abort if arguments are missing. This may happen with redis arguments storage
|
390
|
-
# if Cloud Tasks times out on a job but the job still succeeds
|
391
|
-
if job_args.empty? && [0, -1].exclude?(method(:perform).arity)
|
392
|
-
raise(MissingWorkerArgumentsError, 'worker arguments are missing')
|
393
|
-
end
|
394
|
-
|
395
422
|
# Perform the job
|
396
423
|
perform(*job_args)
|
397
424
|
rescue StandardError => e
|
@@ -14,12 +14,6 @@ module Cloudtasker
|
|
14
14
|
# payloads in Redis
|
15
15
|
REDIS_PAYLOAD_NAMESPACE = 'payload'
|
16
16
|
|
17
|
-
# Arg payload cache keys get expired instead of deleted
|
18
|
-
# in case jobs are re-processed due to connection interruption
|
19
|
-
# (job is successful but Cloud Task considers it as failed due
|
20
|
-
# to network interruption)
|
21
|
-
ARGS_PAYLOAD_CLEANUP_TTL = 3600 # 1 hour
|
22
|
-
|
23
17
|
#
|
24
18
|
# Return a namespaced key
|
25
19
|
#
|
@@ -100,16 +94,13 @@ module Cloudtasker
|
|
100
94
|
# Yied worker
|
101
95
|
resp = yield(worker)
|
102
96
|
|
103
|
-
#
|
104
|
-
|
105
|
-
# succeeds but is considered as failed by Cloud Task due to network interruption.
|
106
|
-
# In such case the job is likely to be re-processed soon after.
|
107
|
-
redis.expire(args_payload_key, ARGS_PAYLOAD_CLEANUP_TTL) if args_payload_key && !worker.job_reenqueued
|
97
|
+
# Delete stored args payload if job has completed
|
98
|
+
redis.del(args_payload_key) if args_payload_key && !worker.job_reenqueued
|
108
99
|
|
109
100
|
resp
|
110
|
-
rescue DeadWorkerError
|
101
|
+
rescue DeadWorkerError => e
|
111
102
|
# Delete stored args payload if job is dead
|
112
|
-
redis.
|
103
|
+
redis.del(args_payload_key) if args_payload_key
|
113
104
|
log_execution_error(worker, e)
|
114
105
|
raise(e)
|
115
106
|
rescue StandardError => e
|
@@ -165,6 +156,7 @@ module Cloudtasker
|
|
165
156
|
},
|
166
157
|
body: worker_payload.to_json
|
167
158
|
},
|
159
|
+
dispatch_deadline: worker.dispatch_deadline.to_i,
|
168
160
|
queue: worker.job_queue
|
169
161
|
}
|
170
162
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: cloudtasker
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.12.
|
4
|
+
version: 0.12.rc9
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Arnaud Lachaume
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2021-
|
11
|
+
date: 2021-04-19 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: activesupport
|