cloudtasker 0.12.rc6 → 0.12.rc11

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 00be8d2e572e3129ef5330cdbb555aa0485edd15b191d5d8ccc5d28a98a0eab8
-  data.tar.gz: 79067d7953556f03e9a81ee9d977cf2734a9843df8f9a4613dbd8d81274bc599
+  metadata.gz: 1c7677889e31eccaadb3cf019f168e2d7133f10682379d0c2ccf1987da450105
+  data.tar.gz: 66ea1098cfbfdc9e5261dba0d6d033e44c8d84ee7623c464fc3bccd3999e708e
 SHA512:
-  metadata.gz: 99d9b7ba99390e6bc10e9ed917262782a7c47814df6412c847dedd06a8a06a6b9ad4eaf5ddbcde03534ec88f34fcb1cda03621a3c2a0202031d087e6cd915c5f
-  data.tar.gz: e15a6a167f15dbd8c390c7d393a80297b6cd805c85d4e5ca16f961ff147c909bebbfcf9c326708e7d7508caa3ff47b726f071e618e3979a1340848420311ddbb
+  metadata.gz: b7ccd9a1e2950d57f51dad86c847373c29caf753b092711cc9be167a8574cdaaec7eeff99dfe08fc3c0e29c4af1a7a21b04e92f36fcc6769b9e3b8f1561c6231
+  data.tar.gz: 7fc85d30cab6c80e8e64afa3285a50dfafb47f66dc214d228c758b7e19481239cf5900fad3bbbd29c3220658529eeeb106f12daf9e10f52a10f4d1bb10864354

data/.rubocop.yml

@@ -12,7 +12,7 @@ Metrics/ModuleLength:
   Max: 150
 
 Metrics/AbcSize:
-  Max: 20
+  Max: 25
   Exclude:
     - 'spec/support/*'
 

data/CHANGELOG.md

@@ -1,17 +1,26 @@
 # Changelog
 
-## Latest RC [v0.12.rc6](https://github.com/keypup-io/cloudtasker/tree/v0.12.rc6) (2021-03-31)
+## Latest RC [v0.12.rc11](https://github.com/keypup-io/cloudtasker/tree/v0.12.rc11) (2021-06-26)
 
-[Full Changelog](https://github.com/keypup-io/cloudtasker/compare/v0.11.0...v0.12.rc6)
+[Full Changelog](https://github.com/keypup-io/cloudtasker/compare/v0.11.0...v0.12.rc11)
 
 **Improvements:**
 - ActiveJob: do not double log errors (ActiveJob has its own error logging)
-- Error logging: Use worker logger so as to include context (job args etc.)
-- Error logging: Do not log exception and stack trace separately, combine them instead.
 - Batch callbacks: Retry jobs when completion callback fails
-- Redis: Use Redis Sets instead of key pattern matching for listing methods (Cron jobs and Local Server)
+- Batch state: use native Redis hashes to store batch state instead of a serialized hash in a string key
 - Batch progress: restrict calculation to direct children by default. Allow depth to be specified. Calculating progress using all tree jobs created significant delays on large batches.
+- Batch redis usage: cleanup batches as they get completed or become dead to avoid excessive redis usage with large batches.
+- Batch expansion: Inject `parent_batch` in jobs. Can be used to expand the parent batch the job is in.
+- Configuration: allow configuration of Cloud Tasks `dispatch deadline` at global and worker level
+- Configuration: allow specifying global `on_error` and `on_dead` callbacks for error reporting
+- Cron jobs: Use Redis Sets instead of key pattern matching for resource listing
+- Error logging: Use worker logger so as to include context (job args etc.)
+- Error logging: Do not log exception and stack trace separately, combine them instead.
+- Local server: Use Redis Sets instead of key pattern matching for resource listing
+- Local server: Guard against nil tasks to prevent job daemon failures
+- Performance: remove use of redis locks and rely on atomic transactions instead for Batch and Unique Job.
 - Worker: raise DeadWorkerError instead of MissingWorkerArgumentsError when arguments are missing. This is more consistent with what middlewares expect.
+- Worker redis usage: delete redis payload storage once the job is successful or dead instead of expiring the key.
 
 **Fixed bugs:**
 - Retries: Enforce job retry limit on job processing. There was an edge case where jobs could be retried indefinitely on batch callback errors.

data/README.md

@@ -35,8 +35,10 @@ A local processing server is also available for development. This local server p
     2. [Logging context](#logging-context)
 9. [Error Handling](#error-handling)
     1. [HTTP Error codes](#http-error-codes)
-    2. [Error callbacks](#error-callbacks)
-    3. [Max retries](#max-retries)
+    2. [Worker callbacks](#worker-callbacks)
+    3. [Global callbacks](#global-callbacks)
+    4. [Max retries](#max-retries)
+    5. [Dispatch deadline](#dispatch-deadline)
 10. [Testing](#testing)
     1. [Test helper setup](#test-helper-setup)
     2. [In-memory queues](#in-memory-queues)
@@ -351,6 +353,53 @@ Cloudtasker.configure do |config|
   #
   # Store all job payloads in Redis exceeding 50 KB:
   # config.store_payloads_in_redis = 50
+
+  #
+  # Specify the dispatch deadline for jobs in Cloud Tasks, in seconds.
+  # Jobs taking longer will be retried by Cloud Tasks, even if they eventually
+  # complete on the server side.
+  #
+  # Note that this option is applied when jobs are enqueued job. Changing this value
+  # will not impact already enqueued jobs.
+  #
+  # This option can also be configured on a per worker basis via
+  # the cloudtasker_options directive.
+  #
+  # Supported since: v0.12.rc8
+  #
+  # Default: 600 seconds (10 minutes)
+  #
+  # config.dispatch_deadline = 600
+
+  #
+  # Specify a proc to be invoked every time a job fails due to a runtime
+  # error.
+  #
+  # This hook is not invoked for DeadWorkerError. See on_dead instead.
+  #
+  # This is useful when you need to apply general exception handling, such
+  # as reporting errors to a third-party service like Rollbar or Bugsnag.
+  #
+  # Note: the worker argument might be nil, such as when InvalidWorkerError is raised.
+  #
+  # Supported since: v0.12.rc11
+  # 
+  # Default: no operation
+  #
+  # config.on_error = ->(error, worker) { Rollbar.error(error) }
+
+  #
+  # Specify a proc to be invoked every time a job dies due to too many
+  # retries.
+  #
+  # This is useful when you need to apply general exception handling, such
+  # logging specific messages/context when a job dies.
+  #
+  # Supported since: v0.12.rc11
+  # 
+  # Default: no operation
+  #
+  # config.on_dead = ->(error, worker) { Rollbar.error(error) }
 end
 ```
 
@@ -631,7 +680,7 @@ Jobs failing will automatically return the following HTTP error code to Cloud Ta
 | 404 | The job has specified an incorrect worker class.  |
 | 422 | An error happened during the execution of the worker (`perform` method) |
 
-### Error callbacks
+### Worker callbacks
 
 Workers can implement the `on_error(error)` and `on_dead(error)` callbacks to do things when a job fails during its execution:
 
@@ -659,6 +708,25 @@ class HandleErrorWorker
 end
 ```
 
+### Global callbacks
+**Supported since**: `0.12.rc11`  
+
+If you need to apply general exception handling logic to your workers you can specify `on_error` and `on_dead` hooks in the Cloudtasker configuration.
+
+This is useful if you need to report errors to third-party services such as Rollbar or Bugsnag.
+
+```ruby
+# config/initializers/cloudtasker.rb
+
+Cloudtasker.configure do |config|
+  #
+  # Report runtime and dead worker errors to Rollbar
+  #
+  config.on_error = -> (error, _worker) { Rollbar.error(error) }
+  config.on_dead = -> (error, _worker) { Rollbar.error(error) }
+end
+```
+
 ### Max retries
 
 By default jobs are retried 25 times - using an exponential backoff - before being declared dead. This number of retries can be customized locally on workers and/or globally via the Cloudtasker initializer.
@@ -721,6 +789,48 @@ class SomeErrorWorker
 end
 ```
 
+### Dispatch deadline
+**Supported since**: `0.12.rc8`  
+
+By default Cloud Tasks will automatically timeout your jobs after 10 minutes, independently of your server HTTP timeout configuration.
+
+You can modify the dispatch deadline for jobs at a global level or on a per job basis.
+
+E.g. Set the default dispatch deadline to 20 minutes.
+```ruby
+# config/initializers/cloudtasker.rb
+
+Cloudtasker.configure do |config|
+  #
+  # Specify the dispatch deadline for jobs in Cloud Tasks, in seconds. 
+  # Jobs taking longer will be retried by Cloud Tasks, even if they eventually
+  # complete on the server side.
+  #
+  # Note that this option is applied when jobs are enqueued job. Changing this value
+  # will not impact already enqueued jobs.
+  #
+  # Default: 600 (10 minutes)
+  #
+  config.dispatch_deadline = 20 * 60 # 20 minutes
+end
+```
+
+E.g. Set a dispatch deadline of 5 minutes on a specific worker
+```ruby
+# app/workers/some_error_worker.rb
+
+class SomeFasterWorker
+  include Cloudtasker::Worker
+
+  # This will override the global setting
+  cloudtasker_options dispatch_deadline: 5 * 60
+
+  def perform
+    # ... do things ...
+  end
+end
+```
+
 ## Testing
 Cloudtasker provides several options to test your workers.
 

data/docs/BATCH_JOBS.md

@@ -18,7 +18,7 @@ Cloudtasker.configure do |config|
 end
 ```
 
-## Example
+## Example: Creating a new batch
 
 The following example defines a worker that adds itself to the batch with different arguments then monitors the success of the batch.
 
@@ -47,6 +47,38 @@ class BatchWorker
 end
 ```
 
+## Example: Expanding the parent batch
+**Note**: `parent_batch` is available since `0.12.rc10`
+
+```ruby
+# All the jobs will be attached to the top parent batch.
+class BatchWorker
+  include Cloudtasker::Worker
+
+  def perform(level, instance)
+    # Use existing parent_batch or create a new one
+    current_batch = parent_batch || batch
+
+    3.times { |n| current_batch.add(self.class, level + 1, n) } if level < 2
+  end
+
+  # Invoked when any descendant (e.g. sub-sub job) is complete
+  def on_batch_node_complete(child)
+    logger.info("Direct or Indirect child complete: #{child.job_id}")
+  end
+
+  # Invoked when a direct descendant is complete
+  def on_child_complete(child)
+    logger.info("Direct child complete: #{child.job_id}")
+  end
+
+  # Invoked when all chidren have finished
+  def on_batch_complete
+    Rails.logger.info("Batch complete")
+  end
+end
+```
+
 ## Available callbacks
 
 The following callbacks are available on your workers to track the progress of the batch:

data/lib/cloudtasker/backend/memory_task.rb

@@ -113,7 +113,7 @@ module Cloudtasker
       # @param [Hash] http_request The HTTP request content.
       # @param [Integer] schedule_time When to run the task (Unix timestamp)
       #
-      def initialize(id:, http_request:, schedule_time: nil, queue: nil, job_retries: 0)
+      def initialize(id:, http_request:, schedule_time: nil, queue: nil, job_retries: 0, **_xargs)
         @id = id
         @http_request = http_request
         @schedule_time = Time.at(schedule_time || 0)

data/lib/cloudtasker/backend/redis_task.rb

@@ -7,7 +7,7 @@ module Cloudtasker
   module Backend
     # Manage local tasks pushed to Redis
     class RedisTask
-      attr_reader :id, :http_request, :schedule_time, :retries, :queue
+      attr_reader :id, :http_request, :schedule_time, :retries, :queue, :dispatch_deadline
 
       RETRY_INTERVAL = 20 # seconds
 
@@ -39,7 +39,7 @@ module Cloudtasker
       def self.all
         if redis.exists?(key)
           # Use Schedule Set if available
-          redis.smembers(key).map { |id| find(id) }
+          redis.smembers(key).map { |id| find(id) }.compact
         else
           # Fallback to redis key matching and migrate tasks
           # to use Task Set instead.
@@ -123,13 +123,15 @@ module Cloudtasker
       # @param [Hash] http_request The HTTP request content.
       # @param [Integer] schedule_time When to run the task (Unix timestamp)
       # @param [Integer] retries The number of times the job failed.
+      # @param [Integer] dispatch_deadline The dispatch_deadline in seconds.
       #
-      def initialize(id:, http_request:, schedule_time: nil, retries: 0, queue: nil)
+      def initialize(id:, http_request:, schedule_time: nil, retries: 0, queue: nil, dispatch_deadline: nil)
         @id = id
         @http_request = http_request
         @schedule_time = Time.at(schedule_time || 0)
         @retries = retries || 0
-        @queue = queue || Cloudtasker::Config::DEFAULT_JOB_QUEUE
+        @queue = queue || Config::DEFAULT_JOB_QUEUE
+        @dispatch_deadline = dispatch_deadline || Config::DEFAULT_DISPATCH_DEADLINE
       end
 
       #
@@ -152,7 +154,8 @@ module Cloudtasker
           http_request: http_request,
           schedule_time: schedule_time.to_i,
           retries: retries,
-          queue: queue
+          queue: queue,
+          dispatch_deadline: dispatch_deadline
         }
       end
 
@@ -176,7 +179,8 @@ module Cloudtasker
           retries: is_error ? retries + 1 : retries,
           http_request: http_request,
           schedule_time: (Time.now + interval).to_i,
-          queue: queue
+          queue: queue,
+          dispatch_deadline: dispatch_deadline
         )
         redis.sadd(self.class.key, id)
       end
@@ -207,6 +211,13 @@ module Cloudtasker
         end
 
         resp
+      rescue Net::ReadTimeout
+        retry_later(RETRY_INTERVAL)
+        Cloudtasker.logger.info(
+          format_log_message(
+            "Task deadline exceeded (#{dispatch_deadline}s) - Retry in #{RETRY_INTERVAL} seconds..."
+          )
+        )
       end
 
       #
@@ -242,7 +253,7 @@ module Cloudtasker
         @http_client ||=
           begin
             uri = URI(http_request[:url])
-            Net::HTTP.new(uri.host, uri.port).tap { |e| e.read_timeout = 60 * 10 }
+            Net::HTTP.new(uri.host, uri.port).tap { |e| e.read_timeout = dispatch_deadline }
           end
       end
 

data/lib/cloudtasker/batch/extension/worker.rb

@@ -6,7 +6,7 @@ module Cloudtasker
       # Include batch related methods onto Cloudtasker::Worker
       # See: Cloudtasker::Batch::Middleware#configure
       module Worker
-        attr_accessor :batch
+        attr_accessor :batch, :parent_batch
       end
     end
   end

data/lib/cloudtasker/batch/job.rb

@@ -17,6 +17,10 @@ module Cloudtasker
       # because the jobs will be either retried or dropped
       IGNORED_ERRORED_CALLBACKS = %i[on_child_error on_child_dead].freeze
 
+      # The maximum number of seconds to wait for a batch state lock
+      # to be acquired.
+      BATCH_MAX_LOCK_WAIT = 60
+
       #
       # Return the cloudtasker redis client
       #
@@ -69,8 +73,12 @@ module Cloudtasker
         # Load extension if not loaded already on the worker class
         worker.class.include(Extension::Worker) unless worker.class <= Extension::Worker
 
-        # Add batch capability
+        # Add batch and parent batch to worker
         worker.batch = new(worker)
+        worker.parent_batch = worker.batch.parent_batch
+
+        # Return the batch
+        worker.batch
       end
 
       #
@@ -176,7 +184,9 @@ module Cloudtasker
       # @return [Hash] The state  of each child worker.
       #
       def batch_state
-        redis.fetch(batch_state_gid)
+        migrate_batch_state_to_redis_hash
+
+        redis.hgetall(batch_state_gid)
       end
 
       #
@@ -208,6 +218,24 @@ module Cloudtasker
         )
       end
 
+      #
+      # This method migrates the batch state to be a Redis hash instead
+      # of a hash stored in a string key.
+      #
+      def migrate_batch_state_to_redis_hash
+        return unless redis.type(batch_state_gid) == 'string'
+
+        # Migrate batch state to Redis hash if it is still using a legacy string key
+        # We acquire a lock then check again
+        redis.with_lock(batch_state_gid, max_wait: BATCH_MAX_LOCK_WAIT) do
+          if redis.type(batch_state_gid) == 'string'
+            state = redis.fetch(batch_state_gid)
+            redis.del(batch_state_gid)
+            redis.hset(batch_state_gid, state) if state.any?
+          end
+        end
+      end
+
       #
       # Save the batch.
       #
@@ -218,8 +246,11 @@ module Cloudtasker
         # complete (success or failure).
         redis.write(batch_gid, worker.to_h)
 
+        # Stop there if no jobs to save
+        return if jobs.empty?
+
         # Save list of child workers
-        redis.write(batch_state_gid, jobs.map { |e| [e.job_id, 'scheduled'] }.to_h)
+        redis.hset(batch_state_gid, jobs.map { |e| [e.job_id, 'scheduled'] }.to_h)
       end
 
       #
@@ -228,29 +259,23 @@ module Cloudtasker
       # @param [String] job_id The batch id.
       # @param [String] status The status of the sub-batch.
       #
-      # @return [<Type>] <description>
-      #
       def update_state(batch_id, status)
-        redis.with_lock(batch_state_gid) do
-          state = batch_state
-          state[batch_id.to_sym] = status.to_s if state.key?(batch_id.to_sym)
-          redis.write(batch_state_gid, state)
-        end
+        migrate_batch_state_to_redis_hash
+
+        # Update the batch state batch_id entry with the new status
+        redis.hset(batch_state_gid, batch_id, status) if redis.hexists(batch_state_gid, batch_id)
       end
 
       #
       # Return true if all the child workers have completed.
       #
-      # @return [<Type>] <description>
+      # @return [Boolean] True if the batch is complete.
       #
       def complete?
-        redis.with_lock(batch_state_gid) do
-          state = redis.fetch(batch_state_gid)
-          return true unless state
+        migrate_batch_state_to_redis_hash
 
-          # Check that all children are complete
-          state.values.all? { |e| COMPLETION_STATUSES.include?(e) }
-        end
+        # Check that all child jobs have completed
+        redis.hvals(batch_state_gid).all? { |e| COMPLETION_STATUSES.include?(e) }
       end
 
       #
@@ -285,8 +310,8 @@ module Cloudtasker
         # Propagate event
         parent_batch&.on_child_complete(self, status)
 
-        # The batch tree is complete. Cleanup the tree.
-        cleanup unless parent_batch
+        # The batch tree is complete. Cleanup the downstream tree.
+        cleanup
       end
 
       #
@@ -331,11 +356,10 @@ module Cloudtasker
       # Remove all batch and sub-batch keys from Redis.
       #
       def cleanup
-        # Capture batch state
-        state = batch_state
+        migrate_batch_state_to_redis_hash
 
         # Delete child batches recursively
-        state.to_h.keys.each { |id| self.class.find(id)&.cleanup }
+        redis.hkeys(batch_state_gid).each { |id| self.class.find(id)&.cleanup }
 
         # Delete batch redis entries
         redis.del(batch_gid)
@@ -402,8 +426,11 @@ module Cloudtasker
         # Perform job
         yield
 
-        # Save batch (if child worker has been enqueued)
-        setup
+        # Save batch if child jobs added
+        setup if jobs.any?
+
+        # Save parent batch if batch expanded
+        parent_batch&.setup if parent_batch&.jobs&.any?
 
         # Complete batch
         complete(:completed)

data/lib/cloudtasker/cloud_task.rb

@@ -3,7 +3,7 @@
 module Cloudtasker
   # An interface class to manage tasks on the backend (Cloud Task or Redis)
   class CloudTask
-    attr_accessor :id, :http_request, :schedule_time, :retries, :queue
+    attr_accessor :id, :http_request, :schedule_time, :retries, :queue, :dispatch_deadline
 
     #
     # The backend to use for cloud tasks.
@@ -73,12 +73,13 @@ module Cloudtasker
     # @param [Integer] retries The number of times the job failed.
     # @param [String] queue The queue the task is in.
     #
-    def initialize(id:, http_request:, schedule_time: nil, retries: 0, queue: nil)
+    def initialize(id:, http_request:, schedule_time: nil, retries: 0, queue: nil, dispatch_deadline: nil)
       @id = id
       @http_request = http_request
       @schedule_time = schedule_time
       @retries = retries || 0
       @queue = queue
+      @dispatch_deadline = dispatch_deadline
     end
 
     #

data/lib/cloudtasker/config.rb

@@ -7,7 +7,8 @@ module Cloudtasker
   class Config
     attr_accessor :redis, :store_payloads_in_redis
     attr_writer :secret, :gcp_location_id, :gcp_project_id,
-                :gcp_queue_prefix, :processor_path, :logger, :mode, :max_retries
+                :gcp_queue_prefix, :processor_path, :logger, :mode, :max_retries,
+                :dispatch_deadline, :on_error, :on_dead
 
     # Max Cloud Task size in bytes
     MAX_TASK_SIZE = 100 * 1024 # 100 KB
@@ -46,6 +47,14 @@ module Cloudtasker
     DEFAULT_QUEUE_CONCURRENCY = 10
     DEFAULT_QUEUE_RETRIES = -1 # unlimited
 
+    # Job timeout configuration for Cloud Tasks
+    DEFAULT_DISPATCH_DEADLINE = 10 * 60 # 10 minutes
+    MIN_DISPATCH_DEADLINE = 15 # seconds
+    MAX_DISPATCH_DEADLINE = 30 * 60 # 30 minutes
+
+    # Default on_error Proc
+    DEFAULT_ON_ERROR = ->(error, worker) {}
+
     # The number of times jobs will be attempted before declaring them dead.
     #
     # With the default retry configuration (maxDoublings = 16 and minBackoff = 0.100s)
@@ -207,6 +216,16 @@ module Cloudtasker
       @gcp_location_id || DEFAULT_LOCATION_ID
     end
 
+    #
+    # Return the Dispatch deadline duration. Cloud Tasks will timeout the job after
+    # this duration is elapsed.
+    #
+    # @return [Integer] The value in seconds.
+    #
+    def dispatch_deadline
+      @dispatch_deadline || DEFAULT_DISPATCH_DEADLINE
+    end
+
     #
     # Return the secret to use to sign the verification tokens
     # attached to tasks.
@@ -214,11 +233,31 @@ module Cloudtasker
     # @return [String] The cloudtasker secret
     #
     def secret
-      @secret || (
+      @secret ||= (
         defined?(Rails) && Rails.application.credentials&.dig(:secret_key_base)
       ) || raise(StandardError, SECRET_MISSING_ERROR)
     end
 
+    #
+    # Return a Proc invoked whenever a worker runtime error is raised.
+    # See Cloudtasker::WorkerHandler.with_worker_handling
+    #
+    # @return [Proc] A Proc handler
+    #
+    def on_error
+      @on_error || DEFAULT_ON_ERROR
+    end
+
+    #
+    # Return a Proc invoked whenever a worker DeadWorkerError is raised.
+    # See Cloudtasker::WorkerHandler.with_worker_handling
+    #
+    # @return [Proc] A Proc handler
+    #
+    def on_dead
+      @on_dead || DEFAULT_ON_ERROR
+    end
+
     #
     # Return the chain of client middlewares.
     #

data/lib/cloudtasker/local_server.rb

@@ -84,7 +84,7 @@ module Cloudtasker
 
       # Deliver task
       begin
-        Thread.current['task'].deliver
+        Thread.current['task']&.deliver
       rescue Errno::EBADF, Errno::ECONNREFUSED => e
         raise(e) unless Thread.current['attempts'] < 3
 

data/lib/cloudtasker/redis_client.rb

@@ -75,14 +75,18 @@ module Cloudtasker
     #   end
     #
     # @param [String] cache_key The cache key to access.
+    # @param [Integer] max_wait The number of seconds after which the lock will be cleared anyway.
     #
-    def with_lock(cache_key)
+    def with_lock(cache_key, max_wait: nil)
       return nil unless cache_key
 
+      # Set max wait
+      max_wait = (max_wait || LOCK_DURATION).to_i
+
       # Wait to acquire lock
       lock_key = [LOCK_KEY_PREFIX, cache_key].join('/')
       client.with do |conn|
-        sleep(LOCK_WAIT_DURATION) until conn.set(lock_key, true, nx: true, ex: LOCK_DURATION)
+        sleep(LOCK_WAIT_DURATION) until conn.set(lock_key, true, nx: true, ex: max_wait)
       end
 
       # yield content

data/lib/cloudtasker/unique_job/job.rb

@@ -149,25 +149,18 @@ module Cloudtasker
       # if taken by another job.
       #
       def lock!
-        redis.with_lock(unique_gid) do
-          locked_id = redis.get(unique_gid)
+        lock_acquired = redis.set(unique_gid, id, nx: true, ex: lock_ttl)
+        lock_already_acquired = !lock_acquired && redis.get(unique_gid) == id
 
-          # Abort job lock process if lock is already taken by another job
-          raise(LockError, locked_id) if locked_id && locked_id != id
-
-          # Take job lock if the lock is currently free
-          redis.set(unique_gid, id, ex: lock_ttl) unless locked_id
-        end
+        raise(LockError) unless lock_acquired || lock_already_acquired
       end
 
       #
       # Delete the job lock.
       #
       def unlock!
-        redis.with_lock(unique_gid) do
-          locked_id = redis.get(unique_gid)
-          redis.del(unique_gid) if locked_id == id
-        end
+        locked_id = redis.get(unique_gid)
+        redis.del(unique_gid) if locked_id == id
       end
     end
   end

data/lib/cloudtasker/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Cloudtasker
-  VERSION = '0.12.rc6'
+  VERSION = '0.12.rc11'
 end

data/lib/cloudtasker/worker.rb

@@ -167,6 +167,22 @@ module Cloudtasker
       (@job_queue ||= self.class.cloudtasker_options_hash[:queue] || Config::DEFAULT_JOB_QUEUE).to_s
     end
 
+    #
+    # Return the Dispatch deadline duration. Cloud Tasks will timeout the job after
+    # this duration is elapsed.
+    #
+    # @return [Integer] The value in seconds.
+    #
+    def dispatch_deadline
+      @dispatch_deadline ||= [
+        [
+          Config::MIN_DISPATCH_DEADLINE,
+          (self.class.cloudtasker_options_hash[:dispatch_deadline] || Cloudtasker.config.dispatch_deadline).to_i
+        ].max,
+        Config::MAX_DISPATCH_DEADLINE
+      ].min
+    end
+
     #
     # Return the Cloudtasker logger instance.
     #

data/lib/cloudtasker/worker_handler.rb

@@ -14,12 +14,6 @@ module Cloudtasker
     # payloads in Redis
     REDIS_PAYLOAD_NAMESPACE = 'payload'
 
-    # Arg payload cache keys get expired instead of deleted
-    # in case jobs are re-processed due to connection interruption
-    # (job is successful but Cloud Task considers it as failed due
-    # to network interruption)
-    ARGS_PAYLOAD_CLEANUP_TTL = 3600 # 1 hour
-
     #
     # Return a namespaced key
     #
@@ -100,20 +94,19 @@ module Cloudtasker
       # Yied worker
       resp = yield(worker)
 
-      # Schedule args payload deletion after job has been successfully processed
-      # Note: we expire the key instead of deleting it immediately in case the job
-      # succeeds but is considered as failed by Cloud Task due to network interruption.
-      # In such case the job is likely to be re-processed soon after.
-      redis.expire(args_payload_key, ARGS_PAYLOAD_CLEANUP_TTL) if args_payload_key && !worker.job_reenqueued
+      # Delete stored args payload if job has completed
+      redis.del(args_payload_key) if args_payload_key && !worker.job_reenqueued
 
       resp
     rescue DeadWorkerError => e
       # Delete stored args payload if job is dead
-      redis.expire(args_payload_key, ARGS_PAYLOAD_CLEANUP_TTL) if args_payload_key
+      redis.del(args_payload_key) if args_payload_key
       log_execution_error(worker, e)
+      Cloudtasker.config.on_dead.call(e, worker)
       raise(e)
     rescue StandardError => e
       log_execution_error(worker, e)
+      Cloudtasker.config.on_error.call(e, worker)
       raise(e)
     end
 
@@ -165,6 +158,7 @@ module Cloudtasker
           },
           body: worker_payload.to_json
         },
+        dispatch_deadline: worker.dispatch_deadline.to_i,
         queue: worker.job_queue
       }
     end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: cloudtasker
 version: !ruby/object:Gem::Version
-  version: 0.12.rc6
+  version: 0.12.rc11
 platform: ruby
 authors:
 - Arnaud Lachaume
-autorequire: 
+autorequire:
 bindir: exe
 cert_chain: []
-date: 2021-03-31 00:00:00.000000000 Z
+date: 2021-06-26 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport
@@ -416,7 +416,7 @@ metadata:
   homepage_uri: https://github.com/keypup-io/cloudtasker
   source_code_uri: https://github.com/keypup-io/cloudtasker
   changelog_uri: https://github.com/keypup-io/cloudtasker/master/tree/CHANGELOG.md
-post_install_message: 
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -432,7 +432,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: 1.3.1
 requirements: []
 rubygems_version: 3.0.0
-signing_key: 
+signing_key:
 specification_version: 4
 summary: Background jobs for Ruby using Google Cloud Tasks (beta)
 test_files: []