cloudtasker 0.11.rc1 → 0.12.rc2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 1a0759638a4af47fcc26467b93039b3c6355908db5cafd0a20972b7e48649b81
4
- data.tar.gz: a0f76c953bc0f64276f5b10f90300bcf7415dbfe305fac6697e4bd9fb83b53c0
3
+ metadata.gz: 11c4d8d88b554792e888929324e79521426f5484973253a44c50a97849a5842c
4
+ data.tar.gz: 1f10788d0ced509c82eea06ddfbd0b1d47a5731507c8bac5c597f41cc13b2d98
5
5
  SHA512:
6
- metadata.gz: ef16621727a56793623e1c582bac1f6e4971b8ff1af3ee36a3d37b22d239f7bb3504a35b5f1ffc9f250343b2ed5ae888e75b8d583b49487f34dc69eb45cd9d13
7
- data.tar.gz: aea0fa6eb2873f6b99df05c613d89363a3c6103fe814123d25a7508ede1fbf92cf653d10e7d57863d7ec6065d1c96e5c5f3ecf371a75ac925a588693849c320e
6
+ metadata.gz: abe6437f179edd589afb88ed970c7870377c836446ba6b4169ffd60565be15fcf92557dd71965795579f58ba19af63a9225ad55cf8c97bb8ab5612e31d5c8bdf
7
+ data.tar.gz: 23d17b1fade5f62594250b239b07e603bc8d8f0862465dd28ec3efc6b260cb86774e88d3b9aead4704eff1d071a3dda2f35d9d68302c6dcef56a1d8b707caffe
data/.rubocop.yml CHANGED
@@ -13,6 +13,8 @@ Metrics/ModuleLength:
13
13
 
14
14
  Metrics/AbcSize:
15
15
  Max: 20
16
+ Exclude:
17
+ - 'spec/support/*'
16
18
 
17
19
  Metrics/LineLength:
18
20
  Max: 120
@@ -20,6 +22,10 @@ Metrics/LineLength:
20
22
  Metrics/MethodLength:
21
23
  Max: 20
22
24
 
25
+ RSpec/DescribeClass:
26
+ Exclude:
27
+ - 'spec/integration/**/*_spec.rb'
28
+
23
29
  RSpec/ExpectInHook:
24
30
  Enabled: false
25
31
 
@@ -43,4 +49,9 @@ Metrics/ParameterLists:
43
49
  CountKeywordArgs: false
44
50
 
45
51
  RSpec/MessageSpies:
46
- Enabled: false
52
+ Enabled: false
53
+
54
+ RSpec/MultipleExpectations:
55
+ Exclude:
56
+ - 'examples/**/*'
57
+ - 'spec/integration/**/*'
data/CHANGELOG.md CHANGED
@@ -1,5 +1,27 @@
1
1
  # Changelog
2
2
 
3
+ ## Latest RC [v0.12.rc1](https://github.com/keypup-io/cloudtasker/tree/v0.12.rc1) (2021-03-11)
4
+
5
+ [Full Changelog](https://github.com/keypup-io/cloudtasker/compare/v0.11.0...v0.12.rc1)
6
+
7
+ **Improvements:**
8
+ - ActiveJob: do not double log errors (ActiveJob has its own error logging)
9
+ - Error logging: Use worker logger so as to include context (job args etc.)
10
+ - Error logging: Do not log exception and stack trace separately, combine them instead.
11
+ - Batch callbacks: Retry jobs when completion callback fails
12
+ - Redis: Use Redis Sets instead of key pattern matching for listing methods (Cron jobs and Local Server)
13
+
14
+ **Fixed bugs:**
15
+ - Retries: Enforce job retry limit on job processing. There was an edge case where jobs could be retried indefinitely on batch callback errors.
16
+
17
+ ## [v0.11.0](https://github.com/keypup-io/cloudtasker/tree/v0.11.0) (2021-03-11)
18
+
19
+ [Full Changelog](https://github.com/keypup-io/cloudtasker/compare/v0.10.0...v0.11.0)
20
+
21
+ **Improvements:**
22
+ - Worker: drop job (return 205 response) when worker arguments are not available (e.g. arguments were stored in Redis and the latter was flushed)
23
+ - Rails: add ActiveJob adapter (thanks @vovimayhem)
24
+
3
25
  ## [v0.10.1](https://github.com/keypup-io/cloudtasker/tree/v0.10.1) (2020-10-05)
4
26
 
5
27
  [Full Changelog](https://github.com/keypup-io/cloudtasker/compare/v0.10.0...v0.10.1)
data/README.md CHANGED
@@ -12,10 +12,13 @@ Cloudtasker also provides optional modules for running [cron jobs](docs/CRON_JOB
12
12
 
13
13
  A local processing server is also available for development. This local server processes jobs in lieu of Cloud Tasks and allows you to work offline.
14
14
 
15
+ **Maturity**: This gem is production-ready. We at Keypup have already processed millions of jobs using Cloudtasker and all related extensions (cron, batch and unique jobs). We are planning to release the official `v1.0.0` somewhere in 2021, in case we've missed any edge-case bug.
16
+
15
17
  ## Summary
16
18
 
17
19
  1. [Installation](#installation)
18
20
  2. [Get started with Rails](#get-started-with-rails)
21
+ 2. [Get started with Rails & ActiveJob](#get-started-with-rails--activejob)
19
22
  3. [Configuring Cloudtasker](#configuring-cloudtasker)
20
23
  1. [Cloud Tasks authentication & permissions](#cloud-tasks-authentication--permissions)
21
24
  2. [Cloudtasker initializer](#cloudtasker-initializer)
@@ -132,6 +135,95 @@ That's it! Your job was picked up by the Cloudtasker local server and sent for p
132
135
 
133
136
  Now jump to the next section to configure your app to use Google Cloud Tasks as a backend.
134
137
 
138
+ ## Get started with Rails & ActiveJob
139
+ **Note**: ActiveJob is supported since `0.11.0`
140
+ **Note**: Cloudtasker extensions (cron, batch and unique jobs) are not available when using cloudtasker via ActiveJob.
141
+
142
+ Cloudtasker is pre-integrated with ActiveJob. Follow the steps below to get started.
143
+
144
+ Install redis on your machine (this is required by the Cloudtasker local processing server)
145
+ ```bash
146
+ # E.g. using brew
147
+ brew install redis
148
+ ```
149
+
150
+ Add the following initializer
151
+ ```ruby
152
+ # config/initializers/cloudtasker.rb
153
+
154
+ Cloudtasker.configure do |config|
155
+ #
156
+ # Adapt the server port to be the one used by your Rails web process
157
+ #
158
+ config.processor_host = 'http://localhost:3000'
159
+
160
+ #
161
+ # If you do not have any Rails secret_key_base defined, uncomment the following
162
+ # This secret is used to authenticate jobs sent to the processing endpoint
163
+ # of your application.
164
+ #
165
+ # config.secret = 'some-long-token'
166
+ end
167
+ ```
168
+
169
+ Configure ActiveJob to use Cloudtasker. You can also configure ActiveJob per environment via the config/environments/:env.rb files
170
+ ```ruby
171
+ # config/application.rb
172
+
173
+ require_relative 'boot'
174
+ require 'rails/all'
175
+
176
+ Bundler.require(*Rails.groups)
177
+
178
+ module Dummy
179
+ class Application < Rails::Application
180
+ # Initialize configuration defaults for originally generated Rails version.
181
+ config.load_defaults 6.0
182
+
183
+ # Settings in config/environments/* take precedence over those specified here.
184
+ # Application configuration can go into files in config/initializers
185
+ # -- all .rb files in that directory are automatically loaded after loading
186
+ # the framework and any gems in your application.
187
+
188
+ # Use cloudtasker as the ActiveJob backend:
189
+ config.active_job.queue_adapter = :cloudtasker
190
+ end
191
+ end
192
+
193
+ ```
194
+
195
+ Define your first job:
196
+ ```ruby
197
+ # app/jobs/example_job.rb
198
+
199
+ class ExampleJob < ApplicationJob
200
+ queue_as :default
201
+
202
+ def perform(some_arg)
203
+ logger.info("Job run with #{some_arg}. This is working!")
204
+ end
205
+ end
206
+ ```
207
+
208
+ Launch Rails and the local Cloudtasker processing server (or add `cloudtasker` to your foreman config as a `worker` process)
209
+ ```bash
210
+ # In one terminal
211
+ > rails s -p 3000
212
+
213
+ # In another terminal
214
+ > cloudtasker
215
+ ```
216
+
217
+ Open a Rails console and enqueue some jobs
218
+ ```ruby
219
+ # Process job as soon as possible
220
+ ExampleJob.perform_later('foo')
221
+
222
+ # Process job in 60 seconds
223
+ ExampleJob.set(wait: 60).perform_later('foo')
224
+ ```
225
+
226
+
135
227
  ## Configuring Cloudtasker
136
228
 
137
229
  ### Cloud Tasks authentication & permissions
@@ -361,6 +453,8 @@ CriticalWorker.schedule(args: [1], queue: :important)
361
453
  ```
362
454
 
363
455
  ## Extensions
456
+ **Note**: Extensions are not available when using cloudtasker via ActiveJob.
457
+
364
458
  Cloudtasker comes with three optional features:
365
459
  - Cron Jobs [[docs](docs/CRON_JOBS.md)]: Run jobs at fixed intervals.
366
460
  - Batch Jobs [[docs](docs/BATCH_JOBS.md)]: Run jobs in jobs and track completion of the overall batch.
@@ -19,32 +19,19 @@ module Cloudtasker
19
19
  # Process payload
20
20
  WorkerHandler.execute_from_payload!(payload)
21
21
  head :no_content
22
- rescue DeadWorkerError, MissingWorkerArgumentsError => e
22
+ rescue DeadWorkerError, MissingWorkerArgumentsError
23
23
  # 205: job will NOT be retried
24
- log_error(e)
25
24
  head :reset_content
26
- rescue InvalidWorkerError => e
25
+ rescue InvalidWorkerError
27
26
  # 404: Job will be retried
28
- log_error(e)
29
27
  head :not_found
30
- rescue StandardError => e
31
- # 404: Job will be retried
32
- log_error(e)
28
+ rescue StandardError
29
+ # 422: Job will be retried
33
30
  head :unprocessable_entity
34
31
  end
35
32
 
36
33
  private
37
34
 
38
- #
39
- # Log an error via cloudtasker logger.
40
- #
41
- # @param [Exception] e The error to log
42
- #
43
- def log_error(error)
44
- Cloudtasker.logger.error(error)
45
- Cloudtasker.logger.error(error.backtrace.join("\n"))
46
- end
47
-
48
35
  #
49
36
  # Parse the request body and return the actual job
50
37
  # payload.
data/cloudtasker.gemspec CHANGED
@@ -41,6 +41,7 @@ Gem::Specification.new do |spec|
41
41
  spec.add_development_dependency 'github_changelog_generator'
42
42
  spec.add_development_dependency 'rake', '>= 12.3.3'
43
43
  spec.add_development_dependency 'rspec', '~> 3.0'
44
+ spec.add_development_dependency 'rspec-json_expectations', '~> 2.2'
44
45
  spec.add_development_dependency 'rubocop', '0.76.0'
45
46
  spec.add_development_dependency 'rubocop-rspec', '1.37.0'
46
47
  spec.add_development_dependency 'semantic_logger'
data/docs/BATCH_JOBS.md CHANGED
@@ -1,4 +1,4 @@
1
- # Cloudtasker Unique Jobs
1
+ # Cloudtasker Batch Jobs
2
2
 
3
3
  **Note**: this extension requires redis
4
4
 
@@ -0,0 +1,82 @@
1
+ # frozen_string_literal: true
2
+
3
+ # ActiveJob docs: http://guides.rubyonrails.org/active_job_basics.html
4
+ # Example adapters ref: https://github.com/rails/rails/tree/master/activejob/lib/active_job/queue_adapters
5
+
6
+ module ActiveJob
7
+ module QueueAdapters
8
+ # == Cloudtasker adapter for Active Job
9
+ #
10
+ # To use Cloudtasker set the queue_adapter config to +:cloudtasker+.
11
+ #
12
+ # Rails.application.config.active_job.queue_adapter = :cloudtasker
13
+ class CloudtaskerAdapter
14
+ SERIALIZATION_FILTERED_KEYS = [
15
+ 'executions', # Given by the worker at processing
16
+ 'provider_job_id', # Also given by the worker at processing
17
+ 'priority' # Not used
18
+ ].freeze
19
+
20
+ # Enqueues the given ActiveJob instance for execution
21
+ #
22
+ # @param job [ActiveJob::Base] The ActiveJob instance
23
+ #
24
+ # @return [Cloudtasker::CloudTask] The Google Task response
25
+ #
26
+ def enqueue(job)
27
+ build_worker(job).schedule
28
+ end
29
+
30
+ # Enqueues the given ActiveJob instance for execution at a given time
31
+ #
32
+ # @param job [ActiveJob::Base] The ActiveJob instance
33
+ # @param precise_timestamp [Integer] The timestamp at which the job must be executed
34
+ #
35
+ # @return [Cloudtasker::CloudTask] The Google Task response
36
+ #
37
+ def enqueue_at(job, precise_timestamp)
38
+ build_worker(job).schedule(time_at: Time.at(precise_timestamp))
39
+ end
40
+
41
+ private
42
+
43
+ def build_worker(job)
44
+ job_serialization = job.serialize.except(*SERIALIZATION_FILTERED_KEYS)
45
+
46
+ JobWrapper.new(
47
+ job_id: job_serialization.delete('job_id'),
48
+ job_queue: job_serialization.delete('queue_name'),
49
+ job_args: [job_serialization]
50
+ )
51
+ end
52
+
53
+ # == Job Wrapper for the Cloudtasker adapter
54
+ #
55
+ # Executes jobs scheduled by the Cloudtasker ActiveJob adapter
56
+ class JobWrapper #:nodoc:
57
+ include Cloudtasker::Worker
58
+
59
+ # Executes the given serialized ActiveJob call.
60
+ # - See https://api.rubyonrails.org/classes/ActiveJob/Core.html#method-i-serialize
61
+ #
62
+ # @param [Hash] job_serialization The serialized ActiveJob call
63
+ #
64
+ # @return [any] The execution of the ActiveJob call
65
+ #
66
+ def perform(job_serialization, *_extra_options)
67
+ job_executions = job_retries < 1 ? 0 : (job_retries + 1)
68
+
69
+ job_serialization.merge!(
70
+ 'job_id' => job_id,
71
+ 'queue_name' => job_queue,
72
+ 'provider_job_id' => task_id,
73
+ 'executions' => job_executions,
74
+ 'priority' => nil
75
+ )
76
+
77
+ Base.execute job_serialization
78
+ end
79
+ end
80
+ end
81
+ end
82
+ end
@@ -23,14 +23,12 @@ module Cloudtasker
23
23
  #
24
24
  # Return a namespaced key.
25
25
  #
26
- # @param [String, Symbol] val The key to namespace
26
+ # @param [String, Symbol, nil] val The key to namespace
27
27
  #
28
28
  # @return [String] The namespaced key.
29
29
  #
30
- def self.key(val)
31
- return nil if val.nil?
32
-
33
- [to_s.underscore, val.to_s].join('/')
30
+ def self.key(val = nil)
31
+ [to_s.underscore, val].compact.map(&:to_s).join('/')
34
32
  end
35
33
 
36
34
  #
@@ -39,9 +37,17 @@ module Cloudtasker
39
37
  # @return [Array<Cloudtasker::Backend::RedisTask>] All the tasks.
40
38
  #
41
39
  def self.all
42
- redis.search(key('*')).map do |gid|
43
- payload = redis.fetch(gid)
44
- new(payload.merge(id: gid.sub(key(''), '')))
40
+ if redis.exists?(key)
41
+ # Use Schedule Set if available
42
+ redis.smembers(key).map { |id| find(id) }
43
+ else
44
+ # Fallback to redis key matching and migrate tasks
45
+ # to use Task Set instead.
46
+ redis.search(key('*')).map do |gid|
47
+ task_id = gid.sub(key(''), '')
48
+ redis.sadd(key, task_id)
49
+ find(task_id)
50
+ end
45
51
  end
46
52
  end
47
53
 
@@ -82,6 +88,7 @@ module Cloudtasker
82
88
 
83
89
  # Save job
84
90
  redis.write(key(id), payload)
91
+ redis.sadd(key, id)
85
92
  new(payload.merge(id: id))
86
93
  end
87
94
 
@@ -105,6 +112,7 @@ module Cloudtasker
105
112
  # @param [String] id The task id.
106
113
  #
107
114
  def self.delete(id)
115
+ redis.srem(key, id)
108
116
  redis.del(key(id))
109
117
  end
110
118
 
@@ -176,7 +184,7 @@ module Cloudtasker
176
184
  # Remove the task from the queue.
177
185
  #
178
186
  def destroy
179
- redis.del(gid)
187
+ self.class.delete(id)
180
188
  end
181
189
 
182
190
  #
@@ -13,6 +13,10 @@ module Cloudtasker
13
13
  # List of statuses triggering a completion callback
14
14
  COMPLETION_STATUSES = %w[completed dead].freeze
15
15
 
16
+ # These callbacks do not need to raise errors on their own
17
+ # because the jobs will be either retried or dropped
18
+ IGNORED_ERRORED_CALLBACKS = %i[on_child_error on_child_dead].freeze
19
+
16
20
  #
17
21
  # Return the cloudtasker redis client
18
22
  #
@@ -62,6 +66,10 @@ module Cloudtasker
62
66
  # @return [Cloudtasker::Batch::Job] The attached batch.
63
67
  #
64
68
  def self.for(worker)
69
+ # Load extension if not loaded already on the worker class
70
+ worker.class.include(Extension::Worker) unless worker.class <= Extension::Worker
71
+
72
+ # Add batch capability
65
73
  worker.batch = new(worker)
66
74
  end
67
75
 
@@ -246,8 +254,8 @@ module Cloudtasker
246
254
  end
247
255
 
248
256
  #
249
- # Run worker callback in a controlled environment to
250
- # avoid interruption of the callback flow.
257
+ # Run worker callback. The error and dead callbacks get
258
+ # silenced should they raise an error.
251
259
  #
252
260
  # @param [String, Symbol] callback The callback to run.
253
261
  # @param [Array<any>] *args The callback arguments.
@@ -257,9 +265,14 @@ module Cloudtasker
257
265
  def run_worker_callback(callback, *args)
258
266
  worker.try(callback, *args)
259
267
  rescue StandardError => e
260
- Cloudtasker.logger.error("Error running callback #{callback}: #{e}")
261
- Cloudtasker.logger.error(e.backtrace.join("\n"))
262
- nil
268
+ # There is no point in retrying jobs due to failure callbacks failing
269
+ # Only completion callbacks will trigger a re-run of the job because
270
+ # these do matter for batch completion
271
+ raise(e) unless IGNORED_ERRORED_CALLBACKS.include?(callback)
272
+
273
+ # Log error instead
274
+ worker.logger.error(e)
275
+ worker.logger.error("Callback #{callback} failed to run. Skipping to preserve error flow.")
263
276
  end
264
277
 
265
278
  #
@@ -271,7 +284,7 @@ module Cloudtasker
271
284
 
272
285
  # Propagate event
273
286
  parent_batch&.on_child_complete(self, status)
274
- ensure
287
+
275
288
  # The batch tree is complete. Cleanup the tree.
276
289
  cleanup unless parent_batch
277
290
  end
@@ -16,6 +16,8 @@ module Cloudtasker
16
16
  Cloudtasker.configure do |config|
17
17
  config.server_middleware { |c| c.add(Middleware::Server) }
18
18
  end
19
+
20
+ # Inject worker extension on main module
19
21
  Cloudtasker::Worker.include(Extension::Worker)
20
22
  end
21
23
  end
@@ -64,6 +64,7 @@ module Cloudtasker
64
64
  return false unless File.exist?('./config/environment.rb')
65
65
 
66
66
  require 'rails'
67
+ require 'cloudtasker/engine'
67
68
  require File.expand_path('./config/environment.rb')
68
69
  end
69
70
 
@@ -21,14 +21,12 @@ module Cloudtasker
21
21
  #
22
22
  # Return a namespaced key.
23
23
  #
24
- # @param [String, Symbol] val The key to namespace
24
+ # @param [String, Symbol, nil] val The key to namespace
25
25
  #
26
26
  # @return [String] The namespaced key.
27
27
  #
28
- def self.key(val)
29
- return nil if val.nil?
30
-
31
- [to_s.underscore, val.to_s].join('/')
28
+ def self.key(val = nil)
29
+ [to_s.underscore, val].compact.map(&:to_s).join('/')
32
30
  end
33
31
 
34
32
  #
@@ -37,8 +35,17 @@ module Cloudtasker
37
35
  # @return [Array<Cloudtasker::Batch::Schedule>] The list of stored schedules.
38
36
  #
39
37
  def self.all
40
- redis.search(key('*')).map do |gid|
41
- find(gid.sub(key(''), ''))
38
+ if redis.exists?(key)
39
+ # Use Schedule Set if available
40
+ redis.smembers(key).map { |id| find(id) }
41
+ else
42
+ # Fallback to redis key matching and migrate schedules
43
+ # to use Schedule Set instead.
44
+ redis.search(key('*')).map do |gid|
45
+ schedule_id = gid.sub(key(''), '')
46
+ redis.sadd(key, schedule_id)
47
+ find(schedule_id)
48
+ end
42
49
  end
43
50
  end
44
51
 
@@ -90,7 +97,7 @@ module Cloudtasker
90
97
  end
91
98
 
92
99
  #
93
- # Destroy a schedule by id.
100
+ # Delete a schedule by id.
94
101
  #
95
102
  # @param [String] id The schedule id.
96
103
  #
@@ -101,6 +108,7 @@ module Cloudtasker
101
108
 
102
109
  # Delete task and stored schedule
103
110
  CloudTask.delete(schedule.task_id) if schedule.task_id
111
+ redis.srem(key, schedule.id)
104
112
  redis.del(schedule.gid)
105
113
  end
106
114
  end
@@ -270,6 +278,7 @@ module Cloudtasker
270
278
 
271
279
  # Save schedule
272
280
  config_was_changed = config_changed?
281
+ redis.sadd(self.class.key, id)
273
282
  redis.write(gid, to_h)
274
283
 
275
284
  # Stop there if backend does not need update
@@ -5,10 +5,14 @@ module Cloudtasker
5
5
  class Engine < ::Rails::Engine
6
6
  isolate_namespace Cloudtasker
7
7
 
8
- initializer 'cloudtasker', before: :load_config_initializers do
8
+ config.before_initialize do
9
+ # Mount cloudtasker processing endpoint
9
10
  Rails.application.routes.append do
10
11
  mount Cloudtasker::Engine, at: '/cloudtasker'
11
12
  end
13
+
14
+ # Add ActiveJob adapter
15
+ require 'active_job/queue_adapters/cloudtasker_adapter' if defined?(::ActiveJob::Railtie)
12
16
  end
13
17
 
14
18
  config.generators do |g|
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Cloudtasker
4
- VERSION = '0.11.rc1'
4
+ VERSION = '0.12.rc2'
5
5
  end
@@ -311,13 +311,25 @@ module Cloudtasker
311
311
  end
312
312
 
313
313
  #
314
- # Return true if the job has excceeded its maximum number
315
- # of retries
314
+ # Return true if the job must declared dead upon raising
315
+ # an error.
316
+ #
317
+ # @return [Boolean] True if the job must die on error.
318
+ #
319
+ def job_must_die?
320
+ job_retries >= job_max_retries
321
+ end
322
+
323
+ #
324
+ # Return true if the job has strictly excceeded its maximum number
325
+ # of retries.
326
+ #
327
+ # Used a preemptive filter when running the job.
316
328
  #
317
329
  # @return [Boolean] True if the job is dead
318
330
  #
319
331
  def job_dead?
320
- job_retries >= job_max_retries
332
+ job_retries > job_max_retries
321
333
  end
322
334
 
323
335
  #
@@ -332,11 +344,35 @@ module Cloudtasker
332
344
  (perform_ended_at - perform_started_at).ceil(3)
333
345
  end
334
346
 
347
+ #
348
+ # Run worker callback.
349
+ #
350
+ # @param [String, Symbol] callback The callback to run.
351
+ # @param [Array<any>] *args The callback arguments.
352
+ #
353
+ # @return [any] The callback return value
354
+ #
355
+ def run_callback(callback, *args)
356
+ try(callback, *args)
357
+ end
358
+
335
359
  #=============================
336
360
  # Private
337
361
  #=============================
338
362
  private
339
363
 
364
+ #
365
+ # Flag the worker as dead by invoking the on_dead hook
366
+ # and raising a DeadWorkerError
367
+ #
368
+ # @param [Exception, nil] error An optional exception to be passed to the DeadWorkerError.
369
+ #
370
+ def flag_as_dead(error = nil)
371
+ run_callback(:on_dead, error || DeadWorkerError.new)
372
+ ensure
373
+ raise(DeadWorkerError, error)
374
+ end
375
+
340
376
  #
341
377
  # Execute the worker perform method through the middleware chain.
342
378
  #
@@ -346,6 +382,9 @@ module Cloudtasker
346
382
  self.perform_started_at = Time.now
347
383
 
348
384
  Cloudtasker.config.server_middleware.invoke(self) do
385
+ # Immediately abort the job if it is already dead
386
+ flag_as_dead if job_dead?
387
+
349
388
  begin
350
389
  # Abort if arguments are missing. This may happen with redis arguments storage
351
390
  # if Cloud Tasks times out on a job but the job still succeeds
@@ -356,12 +395,11 @@ module Cloudtasker
356
395
  # Perform the job
357
396
  perform(*job_args)
358
397
  rescue StandardError => e
359
- try(:on_error, e)
360
- return raise(e) unless job_dead?
398
+ run_callback(:on_error, e)
399
+ return raise(e) unless job_must_die?
361
400
 
362
401
  # Flag job as dead
363
- try(:on_dead, e)
364
- raise(DeadWorkerError, e)
402
+ flag_as_dead(e)
365
403
  end
366
404
  end
367
405
  ensure
@@ -45,6 +45,28 @@ module Cloudtasker
45
45
  end
46
46
  end
47
47
 
48
+ #
49
+ # Log error on execution failure.
50
+ #
51
+ # @param [Cloudtasker::Worker, nil] worker The worker.
52
+ # @param [Exception] error The error to log.
53
+ #
54
+ # @void
55
+ #
56
+ def self.log_execution_error(worker, error)
57
+ # ActiveJob has its own error logging. No need to double log the error.
58
+ # Note: we use string matching instead of class matching as
59
+ # ActiveJob::QueueAdapters::CloudtaskerAdapter::JobWrapper might not be loaded
60
+ return if worker.class.to_s =~ /^ActiveJob::/
61
+
62
+ # Choose logger to use based on context
63
+ # Worker will be nil on InvalidWorkerError - in that case we use generic logging
64
+ logger = worker&.logger || Cloudtasker.logger
65
+
66
+ # Log error
67
+ logger.error(error)
68
+ end
69
+
48
70
  #
49
71
  # Execute a task worker from a task payload
50
72
  #
@@ -88,6 +110,10 @@ module Cloudtasker
88
110
  rescue DeadWorkerError, MissingWorkerArgumentsError => e
89
111
  # Delete stored args payload if job is dead
90
112
  redis.expire(args_payload_key, ARGS_PAYLOAD_CLEANUP_TTL) if args_payload_key
113
+ log_execution_error(worker, e)
114
+ raise(e)
115
+ rescue StandardError => e
116
+ log_execution_error(worker, e)
91
117
  raise(e)
92
118
  end
93
119
 
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: cloudtasker
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.11.rc1
4
+ version: 0.12.rc2
5
5
  platform: ruby
6
6
  authors:
7
7
  - Arnaud Lachaume
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2020-10-16 00:00:00.000000000 Z
11
+ date: 2021-03-12 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: activesupport
@@ -178,6 +178,20 @@ dependencies:
178
178
  - - "~>"
179
179
  - !ruby/object:Gem::Version
180
180
  version: '3.0'
181
+ - !ruby/object:Gem::Dependency
182
+ name: rspec-json_expectations
183
+ requirement: !ruby/object:Gem::Requirement
184
+ requirements:
185
+ - - "~>"
186
+ - !ruby/object:Gem::Version
187
+ version: '2.2'
188
+ type: :development
189
+ prerelease: false
190
+ version_requirements: !ruby/object:Gem::Requirement
191
+ requirements:
192
+ - - "~>"
193
+ - !ruby/object:Gem::Version
194
+ version: '2.2'
181
195
  - !ruby/object:Gem::Dependency
182
196
  name: rubocop
183
197
  requirement: !ruby/object:Gem::Requirement
@@ -343,6 +357,7 @@ files:
343
357
  - gemfiles/semantic_logger_4.7.0.gemfile
344
358
  - gemfiles/semantic_logger_4.7.2.gemfile
345
359
  - gemfiles/semantic_logger_4.7.gemfile
360
+ - lib/active_job/queue_adapters/cloudtasker_adapter.rb
346
361
  - lib/cloudtasker.rb
347
362
  - lib/cloudtasker/authentication_error.rb
348
363
  - lib/cloudtasker/authenticator.rb