cloudtasker 0.6.0 → 0.9.0

data/docs/BATCH_JOBS.md

@@ -2,14 +2,14 @@
 
 **Note**: this extension requires redis
 
-The Cloudtasker batch job extension allows to add sub-jobs to regular jobs. This adds the ability to enqueue a list of jobs and track their overall progression as a groupd of jobs (the batch). This extension allows jobs to define callbacks in their worker to track completion of the batch and take action based on that.
+The Cloudtasker batch job extension allows to add sub-jobs to regular jobs. This adds the ability to enqueue a list of jobs and track their overall progression as a group of jobs (a "batch"). This extension allows jobs to define callbacks in their worker to track completion of the batch and take actions based on that.
 
 ## Configuration
 
 You can enable batch jobs by adding the following to your cloudtasker initializer:
 ```ruby
 # The batch job extension is optional and must be explicitly required
-require 'cloudtasker/batch_job'
+require 'cloudtasker/batch'
 
 Cloudtasker.configure do |config|
   # Specify your redis url.
@@ -53,14 +53,39 @@ The following callbacks are available on your workers to track the progress of t
 
 | Callback | Argument | Description |
 |------|-------------|-----------|
-| `on_batch_node_complete` | `The child job` | Invoked when any descendant (e.g. sub-sub job) successfully completess   |
+| `on_batch_node_complete` | `The child job` | Invoked when any descendant (e.g. sub-sub job) successfully completes   |
 | `on_child_complete` | `The child job` | Invoked when a direct descendant successfully completes   |
 | `on_child_error` | `The child job` | Invoked when a child fails |
 | `on_child_dead` | `The child job` | Invoked when a child has exhausted all of its retries |s
 | `on_batch_complete` | none | Invoked when all chidren have finished or died  |
 
+## Queue management
+
+Jobs added to a batch inherit the queue of the parent. It is possible to specify a different queue when adding a job to a batch using `add_to_queue` batch method.
+
+E.g.
+
+```ruby
+def perform
+  batch.add_to_queue(:critical, SubWorker, arg1, arg2, arg3)
+end
+```
+
 ## Batch completion
 
 Batches complete when all children have successfully completed or died (all retries exhausted).
 
-Jobs that fail in a batch will be retried based on the `max_retries` setting configured globally or on the worker itself. The batch will be considered `pending` while workers retry. Therefore it may be a good idea to reduce the number of retries on your workers using `cloudtasker_options max_retries: 5` to ensure your batches don't hang for too long.
+Jobs that fail in a batch will be retried based on the `max_retries` setting configured globally or on the worker itself. The batch will be considered `pending` while workers retry. Therefore it may be a good idea to reduce the number of retries on your workers using `cloudtasker_options max_retries: 5` to ensure your batches don't hang for too long.
+
+## Batch progress tracking
+
+You can access progression statistics in callback using `batch.progress`. See the [BatchProgress](../lib/cloudtasker/batch/batch_progress.rb) class for more details.
+
+E.g.
+```ruby
+def on_batch_node_complete(_child_job)
+  logger.info("Total: #{batch.progress.total}")
+  logger.info("Completed: #{batch.progress.completed}")
+  logger.info("Progress: #{batch.progress.percent.to_i}%")
+end
+```

data/docs/CRON_JOBS.md

@@ -9,7 +9,7 @@ The Cloudtasker cron job extension allows you to register workers to run at fixe
 You can schedule cron jobs by adding the following to your cloudtasker initializer:
 ```ruby
 # The cron job extension is optional and must be explicitly required
-require 'cloudtasker/cron_job'
+require 'cloudtasker/cron'
 
 Cloudtasker.configure do |config|
   # Specify your redis url.
@@ -18,18 +18,22 @@ Cloudtasker.configure do |config|
 end
 
 # Specify all your cron jobs below. This will synchronize your list of cron jobs (cron jobs previously created and not listed below will be removed).
-Cloudtasker::Cron::Schedule.load_from_hash!(
-  # Run job every minute
-  some_schedule_name: {
-    worker: 'SomeCronWorker',
-    cron: '* * * * *'
-  },
-  # Run job every hour on the fifteenth minute 
-  other_cron_schedule: {
-    worker: 'OtherCronWorker',
-    cron: '15 * * * *'
-  }
-)
+unless Rails.env.test?
+  Cloudtasker::Cron::Schedule.load_from_hash!(
+    # Run job every minute
+    some_schedule_name: {
+      worker: 'SomeCronWorker',
+      cron: '* * * * *'
+    },
+    # Run job every hour on the fifteenth minute 
+    other_cron_schedule: {
+      worker: 'OtherCronWorker',
+      cron: '15 * * * *',
+      queue: 'critical'
+      args: ['foo', 'bar']
+    }
+  )
+end
 ```
 
 ## Using a configuration file
@@ -56,7 +60,7 @@ Then register the jobs inside your Cloudtasker initializer this way:
 # ... Cloudtasker configuration ...
 
 schedule_file = 'config/cloudtasker_cron.yml'
-if File.exist?(schedule_file)
+if File.exist?(schedule_file) && !Rails.env.test?
   Cloudtasker::Cron::Schedule.load_from_hash!(YAML.load_file(schedule_file))
 end
 ```

data/exe/cloudtasker

@@ -3,9 +3,21 @@
 
 require 'bundler/setup'
 require 'cloudtasker/cli'
+require 'optparse'
+
+options = {}
+OptionParser.new do |opts|
+  opts.banner = 'Usage: cloudtasker [options]'
+
+  opts.on('-q QUEUE', '--queue=QUEUE', 'Queue to process and number of threads. ' \
+    "Examples: '-q critical' | '-q critical,2' | '-q critical,3 -q defaults,2'") do |o|
+    options[:queues] ||= []
+    options[:queues] << o.split(',')
+  end
+end.parse!
 
 begin
-  Cloudtasker::CLI.run
+  Cloudtasker::CLI.run(options)
 rescue StandardError => e
   raise e if $DEBUG
 

data/gemfiles/google_cloud_tasks_1.0.gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: ..
   specs:
-    cloudtasker (0.2.0)
+    cloudtasker (0.9.0)
       activesupport
       fugit
       google-cloud-tasks
@@ -82,11 +82,21 @@ GEM
     erubi (1.9.0)
     et-orbi (1.2.2)
       tzinfo
-    faraday (0.17.0)
+    faraday (0.17.1)
       multipart-post (>= 1.2, < 3)
+    faraday-http-cache (2.0.0)
+      faraday (~> 0.8)
     fugit (1.3.3)
       et-orbi (~> 1.1, >= 1.1.8)
       raabro (~> 1.1)
+    github_changelog_generator (1.15.0)
+      activesupport
+      faraday-http-cache
+      multi_json
+      octokit (~> 4.6)
+      rainbow (>= 2.2.1)
+      rake (>= 10.0)
+      retriable (~> 3.0)
     globalid (0.4.2)
       activesupport (>= 4.2.0)
     google-cloud-tasks (1.0.0)
@@ -99,7 +109,7 @@ GEM
       googleauth (~> 0.9)
       grpc (~> 1.24)
       rly (~> 0.2.3)
-    google-protobuf (3.10.1)
+    google-protobuf (3.11.1)
     googleapis-common-protos (1.3.9)
       google-protobuf (~> 3.0)
       googleapis-common-protos-types (~> 1.0)
@@ -124,14 +134,14 @@ GEM
       concurrent-ruby (~> 1.0)
     jaro_winkler (1.5.4)
     jwt (2.2.1)
-    loofah (2.3.1)
+    loofah (2.4.0)
       crass (~> 1.0.2)
       nokogiri (>= 1.5.9)
     mail (2.7.1)
       mini_mime (>= 0.1.1)
     marcel (0.3.3)
       mimemagic (~> 0.3.2)
-    memoist (0.16.1)
+    memoist (0.16.2)
     method_source (0.9.2)
     mimemagic (0.3.3)
     mini_mime (1.0.2)
@@ -140,10 +150,12 @@ GEM
     multi_json (1.14.1)
     multipart-post (2.1.1)
     nio4r (2.5.2)
-    nokogiri (1.10.5)
+    nokogiri (1.10.7)
       mini_portile2 (~> 2.4.0)
+    octokit (4.14.0)
+      sawyer (~> 0.8.0, >= 0.5.3)
     os (1.0.1)
-    parallel (1.19.0)
+    parallel (1.19.1)
     parser (2.6.5.0)
       ast (~> 2.4.0)
     public_suffix (4.0.1)
@@ -180,6 +192,7 @@ GEM
     rainbow (3.0.0)
     rake (10.5.0)
     redis (4.1.3)
+    retriable (3.1.2)
     rly (0.2.3)
     rspec (3.9.0)
       rspec-core (~> 3.9.0)
@@ -209,10 +222,13 @@ GEM
       rainbow (>= 2.2.2, < 4.0)
       ruby-progressbar (~> 1.7)
       unicode-display_width (>= 1.4.0, < 1.7)
-    rubocop-rspec (1.36.0)
+    rubocop-rspec (1.37.0)
       rubocop (>= 0.68.1)
     ruby-progressbar (1.10.1)
     safe_yaml (1.0.5)
+    sawyer (0.8.2)
+      addressable (>= 2.3.5)
+      faraday (> 0.8, < 2.0)
     signet (0.12.0)
       addressable (~> 2.3)
       faraday (~> 0.9)
@@ -239,7 +255,7 @@ GEM
     websocket-driver (0.7.1)
       websocket-extensions (>= 0.1.0)
     websocket-extensions (0.1.4)
-    zeitwerk (2.2.1)
+    zeitwerk (2.2.2)
 
 PLATFORMS
   ruby
@@ -248,6 +264,7 @@ DEPENDENCIES
   appraisal
   bundler (~> 2.0)
   cloudtasker!
+  github_changelog_generator
   google-cloud-tasks (= 1.0)
   rails
   rake (~> 10.0)

data/gemfiles/google_cloud_tasks_1.1.gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: ..
   specs:
-    cloudtasker (0.2.0)
+    cloudtasker (0.9.0)
       activesupport
       fugit
       google-cloud-tasks
@@ -82,11 +82,21 @@ GEM
     erubi (1.9.0)
     et-orbi (1.2.2)
       tzinfo
-    faraday (0.17.0)
+    faraday (0.17.1)
       multipart-post (>= 1.2, < 3)
+    faraday-http-cache (2.0.0)
+      faraday (~> 0.8)
     fugit (1.3.3)
       et-orbi (~> 1.1, >= 1.1.8)
       raabro (~> 1.1)
+    github_changelog_generator (1.15.0)
+      activesupport
+      faraday-http-cache
+      multi_json
+      octokit (~> 4.6)
+      rainbow (>= 2.2.1)
+      rake (>= 10.0)
+      retriable (~> 3.0)
     globalid (0.4.2)
       activesupport (>= 4.2.0)
     google-cloud-tasks (1.1.0)
@@ -99,7 +109,7 @@ GEM
       googleauth (~> 0.9)
       grpc (~> 1.24)
       rly (~> 0.2.3)
-    google-protobuf (3.10.1)
+    google-protobuf (3.11.1)
     googleapis-common-protos (1.3.9)
       google-protobuf (~> 3.0)
       googleapis-common-protos-types (~> 1.0)
@@ -124,14 +134,14 @@ GEM
       concurrent-ruby (~> 1.0)
     jaro_winkler (1.5.4)
     jwt (2.2.1)
-    loofah (2.3.1)
+    loofah (2.4.0)
       crass (~> 1.0.2)
       nokogiri (>= 1.5.9)
     mail (2.7.1)
       mini_mime (>= 0.1.1)
     marcel (0.3.3)
       mimemagic (~> 0.3.2)
-    memoist (0.16.1)
+    memoist (0.16.2)
     method_source (0.9.2)
     mimemagic (0.3.3)
     mini_mime (1.0.2)
@@ -140,10 +150,12 @@ GEM
     multi_json (1.14.1)
     multipart-post (2.1.1)
     nio4r (2.5.2)
-    nokogiri (1.10.5)
+    nokogiri (1.10.7)
       mini_portile2 (~> 2.4.0)
+    octokit (4.14.0)
+      sawyer (~> 0.8.0, >= 0.5.3)
     os (1.0.1)
-    parallel (1.19.0)
+    parallel (1.19.1)
     parser (2.6.5.0)
       ast (~> 2.4.0)
     public_suffix (4.0.1)
@@ -180,6 +192,7 @@ GEM
     rainbow (3.0.0)
     rake (10.5.0)
     redis (4.1.3)
+    retriable (3.1.2)
     rly (0.2.3)
     rspec (3.9.0)
       rspec-core (~> 3.9.0)
@@ -209,10 +222,13 @@ GEM
       rainbow (>= 2.2.2, < 4.0)
       ruby-progressbar (~> 1.7)
       unicode-display_width (>= 1.4.0, < 1.7)
-    rubocop-rspec (1.36.0)
+    rubocop-rspec (1.37.0)
       rubocop (>= 0.68.1)
     ruby-progressbar (1.10.1)
     safe_yaml (1.0.5)
+    sawyer (0.8.2)
+      addressable (>= 2.3.5)
+      faraday (> 0.8, < 2.0)
     signet (0.12.0)
       addressable (~> 2.3)
       faraday (~> 0.9)
@@ -239,7 +255,7 @@ GEM
     websocket-driver (0.7.1)
       websocket-extensions (>= 0.1.0)
     websocket-extensions (0.1.4)
-    zeitwerk (2.2.1)
+    zeitwerk (2.2.2)
 
 PLATFORMS
   ruby
@@ -248,6 +264,7 @@ DEPENDENCIES
   appraisal
   bundler (~> 2.0)
   cloudtasker!
+  github_changelog_generator
   google-cloud-tasks (= 1.1)
   rails
   rake (~> 10.0)

data/gemfiles/google_cloud_tasks_1.2.gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: ..
   specs:
-    cloudtasker (0.2.0)
+    cloudtasker (0.9.0)
       activesupport
       fugit
       google-cloud-tasks
@@ -82,11 +82,21 @@ GEM
     erubi (1.9.0)
     et-orbi (1.2.2)
       tzinfo
-    faraday (0.17.0)
+    faraday (0.17.1)
       multipart-post (>= 1.2, < 3)
+    faraday-http-cache (2.0.0)
+      faraday (~> 0.8)
     fugit (1.3.3)
       et-orbi (~> 1.1, >= 1.1.8)
       raabro (~> 1.1)
+    github_changelog_generator (1.15.0)
+      activesupport
+      faraday-http-cache
+      multi_json
+      octokit (~> 4.6)
+      rainbow (>= 2.2.1)
+      rake (>= 10.0)
+      retriable (~> 3.0)
     globalid (0.4.2)
       activesupport (>= 4.2.0)
     google-cloud-tasks (1.2.0)
@@ -99,7 +109,7 @@ GEM
       googleauth (~> 0.9)
       grpc (~> 1.24)
       rly (~> 0.2.3)
-    google-protobuf (3.10.1-universal-darwin)
+    google-protobuf (3.11.1)
     googleapis-common-protos (1.3.9)
       google-protobuf (~> 3.0)
       googleapis-common-protos-types (~> 1.0)
@@ -113,7 +123,7 @@ GEM
       multi_json (~> 1.11)
       os (>= 0.9, < 2.0)
       signet (~> 0.12)
-    grpc (1.25.0-universal-darwin)
+    grpc (1.25.0)
       google-protobuf (~> 3.8)
       googleapis-common-protos-types (~> 1.0)
     grpc-google-iam-v1 (0.6.9)
@@ -124,14 +134,14 @@ GEM
       concurrent-ruby (~> 1.0)
     jaro_winkler (1.5.4)
     jwt (2.2.1)
-    loofah (2.3.1)
+    loofah (2.4.0)
       crass (~> 1.0.2)
       nokogiri (>= 1.5.9)
     mail (2.7.1)
       mini_mime (>= 0.1.1)
     marcel (0.3.3)
       mimemagic (~> 0.3.2)
-    memoist (0.16.1)
+    memoist (0.16.2)
     method_source (0.9.2)
     mimemagic (0.3.3)
     mini_mime (1.0.2)
@@ -140,10 +150,12 @@ GEM
     multi_json (1.14.1)
     multipart-post (2.1.1)
     nio4r (2.5.2)
-    nokogiri (1.10.5)
+    nokogiri (1.10.7)
       mini_portile2 (~> 2.4.0)
+    octokit (4.14.0)
+      sawyer (~> 0.8.0, >= 0.5.3)
     os (1.0.1)
-    parallel (1.19.0)
+    parallel (1.19.1)
     parser (2.6.5.0)
       ast (~> 2.4.0)
     public_suffix (4.0.1)
@@ -180,6 +192,7 @@ GEM
     rainbow (3.0.0)
     rake (10.5.0)
     redis (4.1.3)
+    retriable (3.1.2)
     rly (0.2.3)
     rspec (3.9.0)
       rspec-core (~> 3.9.0)
@@ -209,10 +222,13 @@ GEM
       rainbow (>= 2.2.2, < 4.0)
       ruby-progressbar (~> 1.7)
       unicode-display_width (>= 1.4.0, < 1.7)
-    rubocop-rspec (1.36.0)
+    rubocop-rspec (1.37.0)
       rubocop (>= 0.68.1)
     ruby-progressbar (1.10.1)
     safe_yaml (1.0.5)
+    sawyer (0.8.2)
+      addressable (>= 2.3.5)
+      faraday (> 0.8, < 2.0)
     signet (0.12.0)
       addressable (~> 2.3)
       faraday (~> 0.9)
@@ -239,7 +255,7 @@ GEM
     websocket-driver (0.7.1)
       websocket-extensions (>= 0.1.0)
     websocket-extensions (0.1.4)
-    zeitwerk (2.2.1)
+    zeitwerk (2.2.2)
 
 PLATFORMS
   ruby
@@ -248,6 +264,7 @@ DEPENDENCIES
   appraisal
   bundler (~> 2.0)
   cloudtasker!
+  github_changelog_generator
   google-cloud-tasks (= 1.2)
   rails
   rake (~> 10.0)