cloudtasker 0.12.rc1 → 0.12.rc6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b89246a4e0d7f658fdb3115c3a00b54df0e55b930428fe52caef471386119906
-  data.tar.gz: 4fe22aa2588e95fffcb223e32e50a2baa5df2730d1e04f8f9d0202aa4138fbd4
+  metadata.gz: 00be8d2e572e3129ef5330cdbb555aa0485edd15b191d5d8ccc5d28a98a0eab8
+  data.tar.gz: 79067d7953556f03e9a81ee9d977cf2734a9843df8f9a4613dbd8d81274bc599
 SHA512:
-  metadata.gz: 1ea18e7601bb394ad4d7f72838bf3eaa9d7ff05df0f0b4f3d08e5d7fa82cee52e51be0c4148d0ac282986f366f6159d0bcc634993f07fbbf28c63ec021d29b5e
-  data.tar.gz: 9282074934628583c6c7b5d2bc6efaf6fac87777bab718adf2204f3eb87143f8bce950711395f6d1ecc6dac6fb349fef52bc94eeb42352322d824b6b35011d5a
+  metadata.gz: 99d9b7ba99390e6bc10e9ed917262782a7c47814df6412c847dedd06a8a06a6b9ad4eaf5ddbcde03534ec88f34fcb1cda03621a3c2a0202031d087e6cd915c5f
+  data.tar.gz: e15a6a167f15dbd8c390c7d393a80297b6cd805c85d4e5ca16f961ff147c909bebbfcf9c326708e7d7508caa3ff47b726f071e618e3979a1340848420311ddbb

data/.rubocop.yml

@@ -6,7 +6,7 @@ AllCops:
     - 'vendor/**/*'
 
 Metrics/ClassLength:
-  Max: 150
+  Max: 200
 
 Metrics/ModuleLength:
   Max: 150

data/CHANGELOG.md

@@ -1,8 +1,8 @@
 # Changelog
 
-## Latest RC [v0.12.rc1](https://github.com/keypup-io/cloudtasker/tree/v0.12.rc1) (2021-03-11)
+## Latest RC [v0.12.rc6](https://github.com/keypup-io/cloudtasker/tree/v0.12.rc6) (2021-03-31)
 
-[Full Changelog](https://github.com/keypup-io/cloudtasker/compare/v0.11.0...v0.12.rc1)
+[Full Changelog](https://github.com/keypup-io/cloudtasker/compare/v0.11.0...v0.12.rc6)
 
 **Improvements:**
 - ActiveJob: do not double log errors (ActiveJob has its own error logging)
@@ -10,6 +10,8 @@
 - Error logging: Do not log exception and stack trace separately, combine them instead.
 - Batch callbacks: Retry jobs when completion callback fails
 - Redis: Use Redis Sets instead of key pattern matching for listing methods (Cron jobs and Local Server)
+- Batch progress: restrict calculation to direct children by default. Allow depth to be specified. Calculating progress using all tree jobs created significant delays on large batches.
+- Worker: raise DeadWorkerError instead of MissingWorkerArgumentsError when arguments are missing. This is more consistent with what middlewares expect.
 
 **Fixed bugs:**
 - Retries: Enforce job retry limit on job processing. There was an edge case where jobs could be retried indefinitely on batch callback errors.

data/README.md

@@ -12,7 +12,7 @@ Cloudtasker also provides optional modules for running [cron jobs](docs/CRON_JOB
 
 A local processing server is also available for development. This local server processes jobs in lieu of Cloud Tasks and allows you to work offline.
 
-**Maturity**: This gem is production-ready. We at Keypup have already processed millions of jobs using Cloudtasker and all related extensions (cron, batch and unique jobs). I'm waiting till the end of 2020 before releasing the official `v1.0.0` in case we've missed any edge-case bug.
+**Maturity**: This gem is production-ready. We at Keypup have already processed millions of jobs using Cloudtasker and all related extensions (cron, batch and unique jobs). We are planning to release the official `v1.0.0` somewhere in 2021, in case we've missed any edge-case bug.
 
 ## Summary
 
@@ -136,7 +136,7 @@ That's it! Your job was picked up by the Cloudtasker local server and sent for p
 Now jump to the next section to configure your app to use Google Cloud Tasks as a backend.
 
 ## Get started with Rails & ActiveJob
-**Note**: ActiveJob is supported since `0.11.0`
+**Note**: ActiveJob is supported since `0.11.0`  
 **Note**: Cloudtasker extensions (cron, batch and unique jobs) are not available when using cloudtasker via ActiveJob.
 
 Cloudtasker is pre-integrated with ActiveJob. Follow the steps below to get started.

data/app/controllers/cloudtasker/worker_controller.rb

@@ -19,7 +19,7 @@ module Cloudtasker
       # Process payload
       WorkerHandler.execute_from_payload!(payload)
       head :no_content
-    rescue DeadWorkerError, MissingWorkerArgumentsError
+    rescue DeadWorkerError
       # 205: job will NOT be retried
       head :reset_content
     rescue InvalidWorkerError

data/docs/BATCH_JOBS.md

@@ -84,8 +84,29 @@ You can access progression statistics in callback using `batch.progress`. See th
 E.g.
 ```ruby
 def on_batch_node_complete(_child_job)
-  logger.info("Total: #{batch.progress.total}")
-  logger.info("Completed: #{batch.progress.completed}")
-  logger.info("Progress: #{batch.progress.percent.to_i}%")
+  progress = batch.progress
+  logger.info("Total: #{progress.total}")
+  logger.info("Completed: #{progress.completed}")
+  logger.info("Progress: #{progress.percent.to_i}%")
+end
+```
+
+**Since:** `v0.12.rc5`  
+By default the `progress` method only considers the direct child jobs to evaluate the batch progress. You can pass `depth: somenumber` to the `progress` method to calculate the actual batch progress in a more granular way. Be careful however that this method recursively calculates progress on the sub-batches and is therefore expensive.
+
+E.g.
+```ruby
+def on_batch_node_complete(_child_job)
+  # Considers the children for batch progress calculation
+  progress_0 = batch.progress # same as batch.progress(depth: 0)
+
+  # Considers the children and grand-children for batch progress calculation
+  progress_1 = batch.progress(depth: 1)
+
+  # Considers the children, grand-children and grand-grand-children for batch progress calculation
+  progress_2 = batch.progress(depth: 3)
+
+  logger.info("Progress: #{progress_1.percent.to_i}%")
+  logger.info("Progress: #{progress_2.percent.to_i}%")
 end
 ```

data/lib/cloudtasker/backend/redis_task.rb

@@ -178,6 +178,7 @@ module Cloudtasker
           schedule_time: (Time.now + interval).to_i,
           queue: queue
         )
+        redis.sadd(self.class.key, id)
       end
 
       #

data/lib/cloudtasker/batch/job.rb

@@ -271,9 +271,8 @@ module Cloudtasker
         raise(e) unless IGNORED_ERRORED_CALLBACKS.include?(callback)
 
         # Log error instead
-        worker.logger.error(
-          ["Callback #{callback} failed to run. Skipping to preserve error flow.", e, e.backtrace].flatten.join("\n")
-        )
+        worker.logger.error(e)
+        worker.logger.error("Callback #{callback} failed to run. Skipping to preserve error flow.")
       end
 
       #
@@ -348,13 +347,20 @@ module Cloudtasker
       #
       # @return [Cloudtasker::Batch::BatchProgress] The batch progress.
       #
-      def progress
+      def progress(depth: 0)
+        depth = depth.to_i
+
         # Capture batch state
         state = batch_state
 
-        # Sum batch progress of current batch and all sub-batches
+        # Return immediately if we do not need to go down the tree
+        return BatchProgress.new(state) if depth <= 0
+
+        # Sum batch progress of current batch and sub-batches up to the specified
+        # depth
         state.to_h.reduce(BatchProgress.new(state)) do |memo, (child_id, child_status)|
-          memo + (self.class.find(child_id)&.progress || BatchProgress.new(child_id => child_status))
+          memo + (self.class.find(child_id)&.progress(depth: depth - 1) ||
+            BatchProgress.new(child_id => child_status))
         end
       end
 

data/lib/cloudtasker/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Cloudtasker
-  VERSION = '0.12.rc1'
+  VERSION = '0.12.rc6'
 end

data/lib/cloudtasker/worker.rb

@@ -332,6 +332,22 @@ module Cloudtasker
       job_retries > job_max_retries
     end
 
+    #
+    # Return true if the job arguments are missing.
+    #
+    # This may happen if a job
+    # was successfully run but retried due to Cloud Task dispatch deadline
+    # exceeded. If the arguments were stored in Redis then they may have
+    # been flushed already after the successful completion.
+    #
+    # If job arguments are missing then the job will simply be declared dead.
+    #
+    # @return [Boolean] True if the arguments are missing.
+    #
+    def arguments_missing?
+      job_args.empty? && [0, -1].exclude?(method(:perform).arity)
+    end
+
     #
     # Return the time taken (in seconds) to perform the job. This duration
     # includes the middlewares and the actual perform method.
@@ -384,14 +400,9 @@ module Cloudtasker
       Cloudtasker.config.server_middleware.invoke(self) do
         # Immediately abort the job if it is already dead
         flag_as_dead if job_dead?
+        flag_as_dead(MissingWorkerArgumentsError.new('worker arguments are missing')) if arguments_missing?
 
         begin
-          # Abort if arguments are missing. This may happen with redis arguments storage
-          # if Cloud Tasks times out on a job but the job still succeeds
-          if job_args.empty? && [0, -1].exclude?(method(:perform).arity)
-            raise(MissingWorkerArgumentsError, 'worker arguments are missing')
-          end
-
           # Perform the job
           perform(*job_args)
         rescue StandardError => e

data/lib/cloudtasker/worker_handler.rb

@@ -64,7 +64,7 @@ module Cloudtasker
       logger = worker&.logger || Cloudtasker.logger
 
       # Log error
-      logger.error([error, error.backtrace].flatten.join("\n"))
+      logger.error(error)
     end
 
     #
@@ -107,7 +107,7 @@ module Cloudtasker
       redis.expire(args_payload_key, ARGS_PAYLOAD_CLEANUP_TTL) if args_payload_key && !worker.job_reenqueued
 
       resp
-    rescue DeadWorkerError, MissingWorkerArgumentsError => e
+    rescue DeadWorkerError => e
       # Delete stored args payload if job is dead
       redis.expire(args_payload_key, ARGS_PAYLOAD_CLEANUP_TTL) if args_payload_key
       log_execution_error(worker, e)

data/lib/cloudtasker/worker_logger.rb

@@ -51,6 +51,26 @@ module Cloudtasker
       Cloudtasker.logger
     end
 
+    #
+    # Format the log message as string.
+    #
+    # @param [Object] msg The log message or object.
+    #
+    # @return [String] The formatted message
+    #
+    def formatted_message_as_string(msg)
+      # Format message
+      msg_content = if msg.is_a?(Exception)
+                      [msg.inspect, msg.backtrace].flatten(1).join("\n")
+                    elsif msg.is_a?(String)
+                      msg
+                    else
+                      msg.inspect
+                    end
+
+      "[Cloudtasker][#{worker.class}][#{worker.job_id}] #{msg_content}"
+    end
+
     #
     # Format main log message.
     #
@@ -59,7 +79,12 @@ module Cloudtasker
     # @return [String] The formatted log message
     #
     def formatted_message(msg)
-      "[Cloudtasker][#{worker.class}][#{worker.job_id}] #{msg}"
+      if msg.is_a?(String)
+        formatted_message_as_string(msg)
+      else
+        # Delegate object formatting to logger
+        msg
+      end
     end
 
     #
@@ -147,7 +172,9 @@ module Cloudtasker
       # ActiveSupport::Logger does not support passing a payload through a block on top
       # of a message.
       if defined?(ActiveSupport::Logger) && logger.is_a?(ActiveSupport::Logger)
-        logger.send(level) { "#{formatted_message(msg)} -- #{payload_block.call}" }
+        # The logger is fairly basic in terms of formatting. All inputs get converted
+        # as regular strings.
+        logger.send(level) { "#{formatted_message_as_string(msg)} -- #{payload_block.call}" }
       else
         logger.send(level, formatted_message(msg), &payload_block)
       end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: cloudtasker
 version: !ruby/object:Gem::Version
-  version: 0.12.rc1
+  version: 0.12.rc6
 platform: ruby
 authors:
 - Arnaud Lachaume
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2021-03-11 00:00:00.000000000 Z
+date: 2021-03-31 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport