cloudtasker 0.12.rc5 → 0.12.rc10

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a1f63a9bfefe90d0cfa0d6b567098ec72efe150894fbd878daa72fe934d27a25
-  data.tar.gz: 347d6358120bd83f116b569fafcd0313a96d8c34a4f6222ca20b0dedda866098
+  metadata.gz: 5763bef46a0c554549150326375c3fb49c9eb77193b6862334dc1da72a5fe34f
+  data.tar.gz: 211272f9129642cb8c7260f639a6a76714161ac37c87dc5f073633a072781b44
 SHA512:
-  metadata.gz: a4ecf9ad17d612133653f70e7691fc746742c3ca62eb8f67c09bdc1992776391e755a5c674b97d0339bd74514958299062610d448b089fe5995dd6abe5756021
-  data.tar.gz: 231f0ba89cf89db4bdb138a1d34d10688161d156c9898da7a067783f75f930b9318a08dfdf240188e23dfa3a0e9dd59b38b1b235b5ab5038eb6727a244877017
+  metadata.gz: ac4012191c2878256abdc446eabc4cd4cf4ff822e426fc7a89e25c77ab2724c5b1d2e508fb88092ddde86680512797c50711d4188798a281cf275b3e03304a0e
+  data.tar.gz: 7a1558eed571862501a3e5a264cfa006014e9454c0f31cb915c9d5229aed6a9c47f539287cb447a226db4302b60d761496256791d9844cc8291164910ec8d6f5

data/.rubocop.yml

@@ -12,7 +12,7 @@ Metrics/ModuleLength:
   Max: 150
 
 Metrics/AbcSize:
-  Max: 20
+  Max: 25
   Exclude:
     - 'spec/support/*'
 

data/CHANGELOG.md

@@ -1,16 +1,25 @@
 # Changelog
 
-## Latest RC [v0.12.rc5](https://github.com/keypup-io/cloudtasker/tree/v0.12.rc5) (2021-03-30)
+## Latest RC [v0.12.rc10](https://github.com/keypup-io/cloudtasker/tree/v0.12.rc10) (2021-05-31)
 
-[Full Changelog](https://github.com/keypup-io/cloudtasker/compare/v0.11.0...v0.12.rc5)
+[Full Changelog](https://github.com/keypup-io/cloudtasker/compare/v0.11.0...v0.12.rc10)
 
 **Improvements:**
 - ActiveJob: do not double log errors (ActiveJob has its own error logging)
-- Error logging: Use worker logger so as to include context (job args etc.)
-- Error logging: Do not log exception and stack trace separately, combine them instead.
 - Batch callbacks: Retry jobs when completion callback fails
-- Redis: Use Redis Sets instead of key pattern matching for listing methods (Cron jobs and Local Server)
+- Batch state: use native Redis hashes to store batch state instead of a serialized hash in a string key
 - Batch progress: restrict calculation to direct children by default. Allow depth to be specified. Calculating progress using all tree jobs created significant delays on large batches.
+- Batch redis usage: cleanup batches as they get completed or become dead to avoid excessive redis usage with large batches.
+- Batch expansion: Inject `parent_batch` in jobs. Can be used to expand the parent batch the job is in.
+- Configuration: allow configuration of Cloud Tasks `dispatch deadline` at global and worker level
+- Cron jobs: Use Redis Sets instead of key pattern matching for resource listing
+- Error logging: Use worker logger so as to include context (job args etc.)
+- Error logging: Do not log exception and stack trace separately, combine them instead.
+- Local server: Use Redis Sets instead of key pattern matching for resource listing
+- Local server: Guard against nil tasks to prevent job daemon failures
+- Performance: remove use of redis locks and rely on atomic transactions instead for Batch and Unique Job.
+- Worker: raise DeadWorkerError instead of MissingWorkerArgumentsError when arguments are missing. This is more consistent with what middlewares expect.
+- Worker redis usage: delete redis payload storage once the job is successful or dead instead of expiring the key.
 
 **Fixed bugs:**
 - Retries: Enforce job retry limit on job processing. There was an edge case where jobs could be retried indefinitely on batch callback errors.

data/README.md

@@ -37,6 +37,7 @@ A local processing server is also available for development. This local server p
     1. [HTTP Error codes](#http-error-codes)
     2. [Error callbacks](#error-callbacks)
     3. [Max retries](#max-retries)
+    4. [Dispatch deadline](#dispatch-deadline)
 10. [Testing](#testing)
     1. [Test helper setup](#test-helper-setup)
     2. [In-memory queues](#in-memory-queues)
@@ -351,6 +352,23 @@ Cloudtasker.configure do |config|
   #
   # Store all job payloads in Redis exceeding 50 KB:
   # config.store_payloads_in_redis = 50
+
+  #
+  # Specify the dispatch deadline for jobs in Cloud Tasks, in seconds.
+  # Jobs taking longer will be retried by Cloud Tasks, even if they eventually
+  # complete on the server side.
+  #
+  # Note that this option is applied when jobs are enqueued job. Changing this value
+  # will not impact already enqueued jobs.
+  #
+  # This option can also be configured on a per worker basis via
+  # the cloudtasker_options directive.
+  #
+  # Supported since: v0.12.rc8
+  #
+  # Default: 600 seconds (10 minutes)
+  #
+  # config.dispatch_deadline = 600
 end
 ```
 
@@ -721,6 +739,48 @@ class SomeErrorWorker
 end
 ```
 
+### Dispatch deadline
+**Supported since**: `0.12.rc8`  
+
+By default Cloud Tasks will automatically timeout your jobs after 10 minutes, independently of your server HTTP timeout configuration.
+
+You can modify the dispatch deadline for jobs at a global level or on a per job basis.
+
+E.g. Set the default dispatch deadline to 20 minutes.
+```ruby
+# config/initializers/cloudtasker.rb
+
+Cloudtasker.configure do |config|
+  #
+  # Specify the dispatch deadline for jobs in Cloud Tasks, in seconds. 
+  # Jobs taking longer will be retried by Cloud Tasks, even if they eventually
+  # complete on the server side.
+  #
+  # Note that this option is applied when jobs are enqueued job. Changing this value
+  # will not impact already enqueued jobs.
+  #
+  # Default: 600 (10 minutes)
+  #
+  config.dispatch_deadline = 20 * 60 # 20 minutes
+end
+```
+
+E.g. Set a dispatch deadline of 5 minutes on a specific worker
+```ruby
+# app/workers/some_error_worker.rb
+
+class SomeFasterWorker
+  include Cloudtasker::Worker
+
+  # This will override the global setting
+  cloudtasker_options dispatch_deadline: 5 * 60
+
+  def perform
+    # ... do things ...
+  end
+end
+```
+
 ## Testing
 Cloudtasker provides several options to test your workers.
 

data/app/controllers/cloudtasker/worker_controller.rb

@@ -19,7 +19,7 @@ module Cloudtasker
       # Process payload
       WorkerHandler.execute_from_payload!(payload)
       head :no_content
-    rescue DeadWorkerError, MissingWorkerArgumentsError
+    rescue DeadWorkerError
       # 205: job will NOT be retried
       head :reset_content
     rescue InvalidWorkerError

data/docs/BATCH_JOBS.md

@@ -18,7 +18,7 @@ Cloudtasker.configure do |config|
 end
 ```
 
-## Example
+## Example: Creating a new batch
 
 The following example defines a worker that adds itself to the batch with different arguments then monitors the success of the batch.
 
@@ -47,6 +47,38 @@ class BatchWorker
 end
 ```
 
+## Example: Expanding the parent batch
+**Note**: `parent_batch` is available since `0.12.rc10`
+
+```ruby
+# All the jobs will be attached to the top parent batch.
+class BatchWorker
+  include Cloudtasker::Worker
+
+  def perform(level, instance)
+    # Use existing parent_batch or create a new one
+    current_batch = parent_batch || batch
+
+    3.times { |n| current_batch.add(self.class, level + 1, n) } if level < 2
+  end
+
+  # Invoked when any descendant (e.g. sub-sub job) is complete
+  def on_batch_node_complete(child)
+    logger.info("Direct or Indirect child complete: #{child.job_id}")
+  end
+
+  # Invoked when a direct descendant is complete
+  def on_child_complete(child)
+    logger.info("Direct child complete: #{child.job_id}")
+  end
+
+  # Invoked when all chidren have finished
+  def on_batch_complete
+    Rails.logger.info("Batch complete")
+  end
+end
+```
+
 ## Available callbacks
 
 The following callbacks are available on your workers to track the progress of the batch:

data/lib/cloudtasker/backend/memory_task.rb

@@ -113,7 +113,7 @@ module Cloudtasker
       # @param [Hash] http_request The HTTP request content.
       # @param [Integer] schedule_time When to run the task (Unix timestamp)
       #
-      def initialize(id:, http_request:, schedule_time: nil, queue: nil, job_retries: 0)
+      def initialize(id:, http_request:, schedule_time: nil, queue: nil, job_retries: 0, **_xargs)
         @id = id
         @http_request = http_request
         @schedule_time = Time.at(schedule_time || 0)

data/lib/cloudtasker/backend/redis_task.rb

@@ -7,7 +7,7 @@ module Cloudtasker
   module Backend
     # Manage local tasks pushed to Redis
     class RedisTask
-      attr_reader :id, :http_request, :schedule_time, :retries, :queue
+      attr_reader :id, :http_request, :schedule_time, :retries, :queue, :dispatch_deadline
 
       RETRY_INTERVAL = 20 # seconds
 
@@ -39,7 +39,7 @@ module Cloudtasker
       def self.all
         if redis.exists?(key)
           # Use Schedule Set if available
-          redis.smembers(key).map { |id| find(id) }
+          redis.smembers(key).map { |id| find(id) }.compact
         else
           # Fallback to redis key matching and migrate tasks
           # to use Task Set instead.
@@ -123,13 +123,15 @@ module Cloudtasker
       # @param [Hash] http_request The HTTP request content.
       # @param [Integer] schedule_time When to run the task (Unix timestamp)
       # @param [Integer] retries The number of times the job failed.
+      # @param [Integer] dispatch_deadline The dispatch_deadline in seconds.
       #
-      def initialize(id:, http_request:, schedule_time: nil, retries: 0, queue: nil)
+      def initialize(id:, http_request:, schedule_time: nil, retries: 0, queue: nil, dispatch_deadline: nil)
         @id = id
         @http_request = http_request
         @schedule_time = Time.at(schedule_time || 0)
         @retries = retries || 0
-        @queue = queue || Cloudtasker::Config::DEFAULT_JOB_QUEUE
+        @queue = queue || Config::DEFAULT_JOB_QUEUE
+        @dispatch_deadline = dispatch_deadline || Config::DEFAULT_DISPATCH_DEADLINE
       end
 
       #
@@ -152,7 +154,8 @@ module Cloudtasker
           http_request: http_request,
           schedule_time: schedule_time.to_i,
           retries: retries,
-          queue: queue
+          queue: queue,
+          dispatch_deadline: dispatch_deadline
         }
       end
 
@@ -176,7 +179,8 @@ module Cloudtasker
           retries: is_error ? retries + 1 : retries,
           http_request: http_request,
           schedule_time: (Time.now + interval).to_i,
-          queue: queue
+          queue: queue,
+          dispatch_deadline: dispatch_deadline
         )
         redis.sadd(self.class.key, id)
       end
@@ -207,6 +211,13 @@ module Cloudtasker
         end
 
         resp
+      rescue Net::ReadTimeout
+        retry_later(RETRY_INTERVAL)
+        Cloudtasker.logger.info(
+          format_log_message(
+            "Task deadline exceeded (#{dispatch_deadline}s) - Retry in #{RETRY_INTERVAL} seconds..."
+          )
+        )
       end
 
       #
@@ -242,7 +253,7 @@ module Cloudtasker
         @http_client ||=
           begin
             uri = URI(http_request[:url])
-            Net::HTTP.new(uri.host, uri.port).tap { |e| e.read_timeout = 60 * 10 }
+            Net::HTTP.new(uri.host, uri.port).tap { |e| e.read_timeout = dispatch_deadline }
           end
       end
 

data/lib/cloudtasker/batch/extension/worker.rb

@@ -6,7 +6,7 @@ module Cloudtasker
       # Include batch related methods onto Cloudtasker::Worker
       # See: Cloudtasker::Batch::Middleware#configure
       module Worker
-        attr_accessor :batch
+        attr_accessor :batch, :parent_batch
       end
     end
   end

data/lib/cloudtasker/batch/job.rb

@@ -17,6 +17,10 @@ module Cloudtasker
       # because the jobs will be either retried or dropped
       IGNORED_ERRORED_CALLBACKS = %i[on_child_error on_child_dead].freeze
 
+      # The maximum number of seconds to wait for a batch state lock
+      # to be acquired.
+      BATCH_MAX_LOCK_WAIT = 60
+
       #
       # Return the cloudtasker redis client
       #
@@ -69,8 +73,12 @@ module Cloudtasker
         # Load extension if not loaded already on the worker class
         worker.class.include(Extension::Worker) unless worker.class <= Extension::Worker
 
-        # Add batch capability
+        # Add batch and parent batch to worker
         worker.batch = new(worker)
+        worker.parent_batch = worker.batch.parent_batch
+
+        # Return the batch
+        worker.batch
       end
 
       #
@@ -176,7 +184,9 @@ module Cloudtasker
       # @return [Hash] The state  of each child worker.
       #
       def batch_state
-        redis.fetch(batch_state_gid)
+        migrate_batch_state_to_redis_hash
+
+        redis.hgetall(batch_state_gid)
       end
 
       #
@@ -208,6 +218,24 @@ module Cloudtasker
         )
       end
 
+      #
+      # This method migrates the batch state to be a Redis hash instead
+      # of a hash stored in a string key.
+      #
+      def migrate_batch_state_to_redis_hash
+        return unless redis.type(batch_state_gid) == 'string'
+
+        # Migrate batch state to Redis hash if it is still using a legacy string key
+        # We acquire a lock then check again
+        redis.with_lock(batch_state_gid, max_wait: BATCH_MAX_LOCK_WAIT) do
+          if redis.type(batch_state_gid) == 'string'
+            state = redis.fetch(batch_state_gid)
+            redis.del(batch_state_gid)
+            redis.hset(batch_state_gid, state) if state.any?
+          end
+        end
+      end
+
       #
       # Save the batch.
       #
@@ -218,8 +246,11 @@ module Cloudtasker
         # complete (success or failure).
         redis.write(batch_gid, worker.to_h)
 
+        # Stop there if no jobs to save
+        return if jobs.empty?
+
         # Save list of child workers
-        redis.write(batch_state_gid, jobs.map { |e| [e.job_id, 'scheduled'] }.to_h)
+        redis.hset(batch_state_gid, jobs.map { |e| [e.job_id, 'scheduled'] }.to_h)
       end
 
       #
@@ -228,29 +259,23 @@ module Cloudtasker
       # @param [String] job_id The batch id.
       # @param [String] status The status of the sub-batch.
       #
-      # @return [<Type>] <description>
-      #
       def update_state(batch_id, status)
-        redis.with_lock(batch_state_gid) do
-          state = batch_state
-          state[batch_id.to_sym] = status.to_s if state.key?(batch_id.to_sym)
-          redis.write(batch_state_gid, state)
-        end
+        migrate_batch_state_to_redis_hash
+
+        # Update the batch state batch_id entry with the new status
+        redis.hset(batch_state_gid, batch_id, status) if redis.hexists(batch_state_gid, batch_id)
       end
 
       #
       # Return true if all the child workers have completed.
       #
-      # @return [<Type>] <description>
+      # @return [Boolean] True if the batch is complete.
       #
       def complete?
-        redis.with_lock(batch_state_gid) do
-          state = redis.fetch(batch_state_gid)
-          return true unless state
+        migrate_batch_state_to_redis_hash
 
-          # Check that all children are complete
-          state.values.all? { |e| COMPLETION_STATUSES.include?(e) }
-        end
+        # Check that all child jobs have completed
+        redis.hvals(batch_state_gid).all? { |e| COMPLETION_STATUSES.include?(e) }
       end
 
       #
@@ -285,8 +310,8 @@ module Cloudtasker
         # Propagate event
         parent_batch&.on_child_complete(self, status)
 
-        # The batch tree is complete. Cleanup the tree.
-        cleanup unless parent_batch
+        # The batch tree is complete. Cleanup the downstream tree.
+        cleanup
       end
 
       #
@@ -331,11 +356,10 @@ module Cloudtasker
       # Remove all batch and sub-batch keys from Redis.
       #
       def cleanup
-        # Capture batch state
-        state = batch_state
+        migrate_batch_state_to_redis_hash
 
         # Delete child batches recursively
-        state.to_h.keys.each { |id| self.class.find(id)&.cleanup }
+        redis.hkeys(batch_state_gid).each { |id| self.class.find(id)&.cleanup }
 
         # Delete batch redis entries
         redis.del(batch_gid)
@@ -402,8 +426,11 @@ module Cloudtasker
         # Perform job
         yield
 
-        # Save batch (if child worker has been enqueued)
-        setup
+        # Save batch if child jobs added
+        setup if jobs.any?
+
+        # Save parent batch if batch expanded
+        parent_batch&.setup if parent_batch&.jobs&.any?
 
         # Complete batch
         complete(:completed)

data/lib/cloudtasker/cloud_task.rb

@@ -3,7 +3,7 @@
 module Cloudtasker
   # An interface class to manage tasks on the backend (Cloud Task or Redis)
   class CloudTask
-    attr_accessor :id, :http_request, :schedule_time, :retries, :queue
+    attr_accessor :id, :http_request, :schedule_time, :retries, :queue, :dispatch_deadline
 
     #
     # The backend to use for cloud tasks.
@@ -73,12 +73,13 @@ module Cloudtasker
     # @param [Integer] retries The number of times the job failed.
     # @param [String] queue The queue the task is in.
     #
-    def initialize(id:, http_request:, schedule_time: nil, retries: 0, queue: nil)
+    def initialize(id:, http_request:, schedule_time: nil, retries: 0, queue: nil, dispatch_deadline: nil)
       @id = id
       @http_request = http_request
       @schedule_time = schedule_time
       @retries = retries || 0
       @queue = queue
+      @dispatch_deadline = dispatch_deadline
     end
 
     #

data/lib/cloudtasker/config.rb

@@ -7,7 +7,7 @@ module Cloudtasker
   class Config
     attr_accessor :redis, :store_payloads_in_redis
     attr_writer :secret, :gcp_location_id, :gcp_project_id,
-                :gcp_queue_prefix, :processor_path, :logger, :mode, :max_retries
+                :gcp_queue_prefix, :processor_path, :logger, :mode, :max_retries, :dispatch_deadline
 
     # Max Cloud Task size in bytes
     MAX_TASK_SIZE = 100 * 1024 # 100 KB
@@ -46,6 +46,11 @@ module Cloudtasker
     DEFAULT_QUEUE_CONCURRENCY = 10
     DEFAULT_QUEUE_RETRIES = -1 # unlimited
 
+    # Job timeout configuration for Cloud Tasks
+    DEFAULT_DISPATCH_DEADLINE = 10 * 60 # 10 minutes
+    MIN_DISPATCH_DEADLINE = 15 # seconds
+    MAX_DISPATCH_DEADLINE = 30 * 60 # 30 minutes
+
     # The number of times jobs will be attempted before declaring them dead.
     #
     # With the default retry configuration (maxDoublings = 16 and minBackoff = 0.100s)
@@ -207,6 +212,16 @@ module Cloudtasker
       @gcp_location_id || DEFAULT_LOCATION_ID
     end
 
+    #
+    # Return the Dispatch deadline duration. Cloud Tasks will timeout the job after
+    # this duration is elapsed.
+    #
+    # @return [Integer] The value in seconds.
+    #
+    def dispatch_deadline
+      @dispatch_deadline || DEFAULT_DISPATCH_DEADLINE
+    end
+
     #
     # Return the secret to use to sign the verification tokens
     # attached to tasks.

data/lib/cloudtasker/local_server.rb

@@ -84,7 +84,7 @@ module Cloudtasker
 
       # Deliver task
       begin
-        Thread.current['task'].deliver
+        Thread.current['task']&.deliver
       rescue Errno::EBADF, Errno::ECONNREFUSED => e
         raise(e) unless Thread.current['attempts'] < 3
 

data/lib/cloudtasker/redis_client.rb

@@ -75,14 +75,18 @@ module Cloudtasker
     #   end
     #
     # @param [String] cache_key The cache key to access.
+    # @param [Integer] max_wait The number of seconds after which the lock will be cleared anyway.
     #
-    def with_lock(cache_key)
+    def with_lock(cache_key, max_wait: nil)
       return nil unless cache_key
 
+      # Set max wait
+      max_wait = (max_wait || LOCK_DURATION).to_i
+
       # Wait to acquire lock
       lock_key = [LOCK_KEY_PREFIX, cache_key].join('/')
       client.with do |conn|
-        sleep(LOCK_WAIT_DURATION) until conn.set(lock_key, true, nx: true, ex: LOCK_DURATION)
+        sleep(LOCK_WAIT_DURATION) until conn.set(lock_key, true, nx: true, ex: max_wait)
       end
 
       # yield content

data/lib/cloudtasker/unique_job/job.rb

@@ -149,25 +149,18 @@ module Cloudtasker
       # if taken by another job.
       #
       def lock!
-        redis.with_lock(unique_gid) do
-          locked_id = redis.get(unique_gid)
+        lock_acquired = redis.set(unique_gid, id, nx: true, ex: lock_ttl)
+        lock_already_acquired = !lock_acquired && redis.get(unique_gid) == id
 
-          # Abort job lock process if lock is already taken by another job
-          raise(LockError, locked_id) if locked_id && locked_id != id
-
-          # Take job lock if the lock is currently free
-          redis.set(unique_gid, id, ex: lock_ttl) unless locked_id
-        end
+        raise(LockError) unless lock_acquired || lock_already_acquired
       end
 
       #
       # Delete the job lock.
       #
       def unlock!
-        redis.with_lock(unique_gid) do
-          locked_id = redis.get(unique_gid)
-          redis.del(unique_gid) if locked_id == id
-        end
+        locked_id = redis.get(unique_gid)
+        redis.del(unique_gid) if locked_id == id
       end
     end
   end

data/lib/cloudtasker/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Cloudtasker
-  VERSION = '0.12.rc5'
+  VERSION = '0.12.rc10'
 end

data/lib/cloudtasker/worker.rb

@@ -167,6 +167,22 @@ module Cloudtasker
       (@job_queue ||= self.class.cloudtasker_options_hash[:queue] || Config::DEFAULT_JOB_QUEUE).to_s
     end
 
+    #
+    # Return the Dispatch deadline duration. Cloud Tasks will timeout the job after
+    # this duration is elapsed.
+    #
+    # @return [Integer] The value in seconds.
+    #
+    def dispatch_deadline
+      @dispatch_deadline ||= [
+        [
+          Config::MIN_DISPATCH_DEADLINE,
+          (self.class.cloudtasker_options_hash[:dispatch_deadline] || Cloudtasker.config.dispatch_deadline).to_i
+        ].max,
+        Config::MAX_DISPATCH_DEADLINE
+      ].min
+    end
+
     #
     # Return the Cloudtasker logger instance.
     #
@@ -332,6 +348,22 @@ module Cloudtasker
       job_retries > job_max_retries
     end
 
+    #
+    # Return true if the job arguments are missing.
+    #
+    # This may happen if a job
+    # was successfully run but retried due to Cloud Task dispatch deadline
+    # exceeded. If the arguments were stored in Redis then they may have
+    # been flushed already after the successful completion.
+    #
+    # If job arguments are missing then the job will simply be declared dead.
+    #
+    # @return [Boolean] True if the arguments are missing.
+    #
+    def arguments_missing?
+      job_args.empty? && [0, -1].exclude?(method(:perform).arity)
+    end
+
     #
     # Return the time taken (in seconds) to perform the job. This duration
     # includes the middlewares and the actual perform method.
@@ -384,14 +416,9 @@ module Cloudtasker
       Cloudtasker.config.server_middleware.invoke(self) do
         # Immediately abort the job if it is already dead
         flag_as_dead if job_dead?
+        flag_as_dead(MissingWorkerArgumentsError.new('worker arguments are missing')) if arguments_missing?
 
         begin
-          # Abort if arguments are missing. This may happen with redis arguments storage
-          # if Cloud Tasks times out on a job but the job still succeeds
-          if job_args.empty? && [0, -1].exclude?(method(:perform).arity)
-            raise(MissingWorkerArgumentsError, 'worker arguments are missing')
-          end
-
           # Perform the job
           perform(*job_args)
         rescue StandardError => e

data/lib/cloudtasker/worker_handler.rb

@@ -14,12 +14,6 @@ module Cloudtasker
     # payloads in Redis
     REDIS_PAYLOAD_NAMESPACE = 'payload'
 
-    # Arg payload cache keys get expired instead of deleted
-    # in case jobs are re-processed due to connection interruption
-    # (job is successful but Cloud Task considers it as failed due
-    # to network interruption)
-    ARGS_PAYLOAD_CLEANUP_TTL = 3600 # 1 hour
-
     #
     # Return a namespaced key
     #
@@ -100,16 +94,13 @@ module Cloudtasker
       # Yied worker
       resp = yield(worker)
 
-      # Schedule args payload deletion after job has been successfully processed
-      # Note: we expire the key instead of deleting it immediately in case the job
-      # succeeds but is considered as failed by Cloud Task due to network interruption.
-      # In such case the job is likely to be re-processed soon after.
-      redis.expire(args_payload_key, ARGS_PAYLOAD_CLEANUP_TTL) if args_payload_key && !worker.job_reenqueued
+      # Delete stored args payload if job has completed
+      redis.del(args_payload_key) if args_payload_key && !worker.job_reenqueued
 
       resp
-    rescue DeadWorkerError, MissingWorkerArgumentsError => e
+    rescue DeadWorkerError => e
       # Delete stored args payload if job is dead
-      redis.expire(args_payload_key, ARGS_PAYLOAD_CLEANUP_TTL) if args_payload_key
+      redis.del(args_payload_key) if args_payload_key
       log_execution_error(worker, e)
       raise(e)
     rescue StandardError => e
@@ -165,6 +156,7 @@ module Cloudtasker
           },
           body: worker_payload.to_json
         },
+        dispatch_deadline: worker.dispatch_deadline.to_i,
         queue: worker.job_queue
       }
     end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: cloudtasker
 version: !ruby/object:Gem::Version
-  version: 0.12.rc5
+  version: 0.12.rc10
 platform: ruby
 authors:
 - Arnaud Lachaume
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2021-03-30 00:00:00.000000000 Z
+date: 2021-05-31 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport