cloudtasker 0.10.rc6 → 0.10.rc7

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 796e69d0470947fe0af6ea530cfd6c5b69e95b6a1be42b307e84b27dd9246198
-  data.tar.gz: 5c51dd25b3033546a7115d83d638123e80a9ee26fb74845b88a02247edb39461
+  metadata.gz: 7c5440326c0fa695e469a341d58b976bd4027179440a607ce34c15c9dd4d697e
+  data.tar.gz: 671c2bcf90c199c68136a14efa62410d17e3a850dad1e6b5bf6a02b083169d75
 SHA512:
-  metadata.gz: 134e6d344e75a9500850b135ea215c2b64d31cf366892acc0aab1b37c47e758dfa7e2bd00015ec6d8bb47556aba002bb578f2365ea34196d3343b24cee0f2156
-  data.tar.gz: 73bd3cbd91fa1938b97df2cd7f5125a3f90205fb31b5d12efe5b64c4c0aa03e74f6f83cd8c9267b95b5553be93395a45db4f365bbb49b5623a97bf982bf9d2d7
+  metadata.gz: f6adad797eedb6ff0d589ea2dfa6b109709f629ad82cf64e9755e7da235b4a4fb902d4c48794981dcd527a6933196686a72fce46b2572912fd84b1a3396bcd33
+  data.tar.gz: 78fa73bafda04c16b46bed2bd8930e5e5ef3badf046f2e665b41d156e9644c20468788a0ffa173fcb6585bdb24a1bdcff599be11f41a5420c1302747e209bc52

data/.rubocop.yml

@@ -40,4 +40,7 @@ Style/Documentation:
     - 'spec/**/*'
 
 Metrics/ParameterLists:
-  CountKeywordArgs: false
+  CountKeywordArgs: false
+
+RSpec/MessageSpies:
+  Enabled: false

data/README.md

@@ -6,11 +6,11 @@ Background jobs for Ruby using Google Cloud Tasks.
 
 Cloudtasker provides an easy to manage interface to Google Cloud Tasks for background job processing. Workers can be defined programmatically using the Cloudtasker DSL and enqueued for processing using a simple to use API.
 
-Cloudtasker is particularly suited for serverless applications only responding to HTTP requests and where running a dedicated job processing is not an option (e.g. deploy via [Cloud Run](https://cloud.google.com/run)). All jobs enqueued in Cloud Tasks via Cloudtasker eventually get processed by your application via HTTP requests.
+Cloudtasker is particularly suited for serverless applications only responding to HTTP requests and where running a dedicated job processing server is not an option (e.g. deploy via [Cloud Run](https://cloud.google.com/run)). All jobs enqueued in Cloud Tasks via Cloudtasker eventually get processed by your application via HTTP requests.
 
 Cloudtasker also provides optional modules for running [cron jobs](docs/CRON_JOBS.md), [batch jobs](docs/BATCH_JOBS.md) and [unique jobs](docs/UNIQUE_JOBS.md).
 
-A local processing server is also available in development. This local server processes jobs in lieu of Cloud Tasks and allows you to work offline.
+A local processing server is also available for development. This local server processes jobs in lieu of Cloud Tasks and allows you to work offline.
 
 ## Summary
 
@@ -34,7 +34,11 @@ A local processing server is also available in development. This local server pr
     1. [HTTP Error codes](#http-error-codes)
     2. [Error callbacks](#error-callbacks)
     3. [Max retries](#max-retries)
-10. [Best practices building workers](#best-practices-building-workers)
+10. [Testing](#testing)
+    1. [Test helper setup](#test-helper-setup)
+    2. [In-memory queues](#in-memory-queues)
+    3. [Unit tests](#unit-tests)
+11. [Best practices building workers](#best-practices-building-workers)
 
 ## Installation
 
@@ -48,7 +52,7 @@ And then execute:
 
     $ bundle
 
-Or install it yourself as:
+Or install it yourself with:
 
     $ gem install cloudtasker
 
@@ -218,7 +222,7 @@ Cloudtasker.configure do |config|
 
   # 
   # Specify how many retries are allowed on jobs. This number of retries excludes any
-  # connectivity error that would be due to the application being down or unreachable.
+  # connectivity error due to the application being down or unreachable.
   # 
   # Default: 25
   # 
@@ -289,7 +293,7 @@ MyWorker.schedule(args: [arg1, arg2], time_at: Time.parse('2025-01-01 00:50:00Z'
 MyWorker.schedule(args: [arg1, arg2], time_in: 5 * 60, queue: 'critical')
 ```
 
-Cloudtasker also provides a helper for re-enqueuing jobs. Re-enqueued jobs keep the same worker id. Some middlewares may rely on this to track the fact that that a job didn't actually complete (e.g. Cloustasker batch). This is optional and you can always fallback to using exception management (raise an error) to retry/re-enqueue jobs.
+Cloudtasker also provides a helper for re-enqueuing jobs. Re-enqueued jobs keep the same job id. Some middlewares may rely on this to track the fact that that a job didn't actually complete (e.g. Cloustasker batch). This is optional and you can always fallback to using exception management (raise an error) to retry/re-enqueue jobs.
 
 E.g.
 ```ruby
@@ -474,7 +478,7 @@ The way contextual information is displayed depends on the logger itself. For ex
 
 Contextual information can be customised globally and locally using a log context_processor. By default the `Cloudtasker::WorkerLogger` is configured the following way:
 ```ruby
-Cloudtasker::WorkerLogger.log_context_processor = ->(worker) { worker.to_h.slice(:worker, :job_id, :job_meta) }
+Cloudtasker::WorkerLogger.log_context_processor = ->(worker) { worker.to_h.slice(:worker, :job_id, :job_meta, :job_queue, :task_id) }
 ```
 
 You can decide to add a global identifier for your worker logs using the following:
@@ -482,7 +486,7 @@ You can decide to add a global identifier for your worker logs using the followi
 # config/initializers/cloudtasker.rb
 
 Cloudtasker::WorkerLogger.log_context_processor = lambda { |worker|
-  worker.to_h.slice(:worker, :job_id, :job_meta).merge(app: 'my-app')
+  worker.to_h.slice(:worker, :job_id, :job_meta, :job_queue, :task_id).merge(app: 'my-app')
 }
 ```
 
@@ -520,7 +524,7 @@ The Google Cloud Task UI (GCP console) lists all the tasks pending/retrying and
 
 ## Error Handling
 
-Jobs failing will automatically return an HTTP error to Cloud Task and trigger a retry at a later time. The number of Cloud Task retries Cloud Task will depend on the configuration of your queue in Cloud Tasks.
+Jobs failures will return an HTTP error to Cloud Task and trigger a retry at a later time. The number of Cloud Task retries depends on the configuration of your queue in Cloud Tasks.
 
 ### HTTP Error codes
 
@@ -528,6 +532,7 @@ Jobs failing will automatically return the following HTTP error code to Cloud Ta
 
 | Code | Description |
 |------|-------------|
+| 204 | The job was processed successfully |
 | 205 | The job is dead and has been removed from the queue |
 | 404 | The job has specified an incorrect worker class.  |
 | 422 | An error happened during the execution of the worker (`perform` method) |
@@ -566,7 +571,7 @@ By default jobs are retried 25 times - using an exponential backoff - before bei
 
 Note that the number of retries set on your Cloud Task queue should be many times higher than the number of retries configured in Cloudtasker because Cloud Task also includes failures to connect to your application. Ideally set the number of retries to `unlimited` in Cloud Tasks.
 
-**Note**: The `X-CloudTasks-TaskExecutionCount` header sent by Google Cloud Tasks and providing the number of retries outside of `HTTP 503` (instance not reachable) is currently bugged and remains at `0` all the time. Starting with `0.10.rc3` Cloudtasker uses the `X-CloudTasks-TaskRetryCount` header to detect the number of retries. This header includes `HTTP 503` errors which means that if your application is down at some point, jobs will fail and these failures will be counted toward the maximum number of retries. A [bug report](https://issuetracker.google.com/issues/154532072) has been raised with GCP to address this issue. Once fixed we will revert to using `X-CloudTasks-TaskExecutionCount` to avoid counting `HTTP 503` as job failures.
+**Note**: The `X-CloudTasks-TaskExecutionCount` header sent by Google Cloud Tasks and providing the number of retries outside of `HTTP 503` (instance not reachable) is currently bugged and remains at `0` all the time. Starting with `v0.10.rc3` Cloudtasker uses the `X-CloudTasks-TaskRetryCount` header to detect the number of retries. This header includes `HTTP 503` errors which means that if your application is down at some point, jobs will fail and these failures will be counted toward the maximum number of retries. A [bug report](https://issuetracker.google.com/issues/154532072) has been raised with GCP to address this issue. Once fixed we will revert to using `X-CloudTasks-TaskExecutionCount` to avoid counting `HTTP 503` as job failures.
 
 E.g. Set max number of retries globally via the cloudtasker initializer.
 ```ruby
@@ -601,6 +606,97 @@ class SomeErrorWorker
 end
 ```
 
+## Testing
+Cloudtasker provides several options to test your workers.
+
+### Test helper setup
+Require `cloudtasker/testing` in your `rails_helper.rb` (Rspec Rails) or `spec_helper.rb` (Rspec) or test unit helper file then enable one of the three modes:
+
+```ruby
+require 'cloudtasker/testing'
+
+# Mode 1 (default): Push jobs to Google Cloud Tasks (env != development) or Redis (env == development)
+Cloudtasker::Testing.enable!
+
+# Mode 2: Push jobs to an in-memory queue. Jobs will not be processed until you call 
+# Cloudtasker::Worker.drain_all (process all jobs) or MyWorker.drain (process jobs for specific worker)
+Cloudtasker::Testing.fake!
+
+# Mode 3: Push jobs to an in-memory queue. Jobs will be processed immediately.
+Cloudtasker::Testing.inline!
+```
+
+You can query the current testing mode with:
+```ruby
+Cloudtasker::Testing.enabled?
+Cloudtasker::Testing.fake?
+Cloudtasker::Testing.inline?
+```
+
+Each testing mode accepts a block argument to temporarily switch to it:
+```ruby
+# Enable fake mode for all tests
+Cloudtasker::Testing.fake!
+
+# Enable inline! mode temporarily for a given test
+Cloudtasker.inline! do
+   MyWorker.perform_async(1,2)
+end
+```
+
+Note that extension middlewares - e.g. unique job, batch job etc. - run in test mode. You can disable middlewares in your tests by adding the following to your test helper:
+```ruby
+# Remove all middlewares
+Cloudtasker.configure do |c|
+  c.client_middleware.clear
+  c.server_middleware.clear
+end
+
+# Remove all unique job middlewares
+Cloudtasker.configure do |c|
+  c.client_middleware.remove(Cloudtasker::UniqueJob::Middleware::Client)
+  c.server_middleware.remove(Cloudtasker::UniqueJob::Middleware::Server)
+end
+```
+
+### In-memory queues
+The `fake!` or `inline!` modes use in-memory queues, which can be queried and controlled using the following methods:
+
+```ruby
+# Perform all jobs in queue
+Cloudtasker::Worker.drain_all
+
+# Remove all jobs in queue
+Cloudtasker::Worker.clear_all
+
+# Perform all jobs in queue for a specific worker type
+MyWorker.drain
+
+# Return the list of jobs in queue for a specific worker type
+MyWorker.jobs
+```
+
+### Unit tests
+Below are examples of rspec tests. It is assumed that `Cloudtasker::Testing.fake!` has been set in the test helper.
+
+**Example 1**: Testing that a job is scheduled
+```ruby
+describe 'worker scheduling'
+  subject(:enqueue_job) { MyWorker.perform_async(1,2) } 
+  
+  it { expect { enqueue_job }.to change(MyWorker.jobs, :size).by(1) } 
+end
+```
+
+**Example 2**: Testing job execution logic
+```ruby
+describe 'worker calls api'
+  subject { Cloudtasker::Testing.inline! { MyApiWorker.perform_async(1,2) } }
+  
+  before { expect(MyApi).to receive(:fetch).and_return([]) }
+  it { is_expected.to be_truthy }
+end
+```
 
 ## Best practices building workers
 

data/lib/cloudtasker/backend/memory_task.rb

@@ -8,6 +8,15 @@ module Cloudtasker
       attr_accessor :job_retries
       attr_reader :id, :http_request, :schedule_time, :queue
 
+      #
+      # Return true if we are in test inline execution mode.
+      #
+      # @return [Boolean] True if inline mode enabled.
+      #
+      def self.inline_mode?
+        defined?(Cloudtasker::Testing) && Cloudtasker::Testing.inline?
+      end
+
       #
       # Return the task queue. A worker class name
       #
@@ -57,7 +66,7 @@ module Cloudtasker
         queue << task
 
         # Execute task immediately if in testing and inline mode enabled
-        task.execute if defined?(Cloudtasker::Testing) && Cloudtasker::Testing.inline?
+        task.execute if inline_mode?
 
         task
       end
@@ -157,8 +166,9 @@ module Cloudtasker
         # Delete task
         self.class.delete(id)
         resp
-      rescue StandardError
+      rescue StandardError => e
         self.job_retries += 1
+        raise(e) if self.class.inline_mode?
       end
 
       #

data/lib/cloudtasker/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Cloudtasker
-  VERSION = '0.10.rc6'
+  VERSION = '0.10.rc7'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: cloudtasker
 version: !ruby/object:Gem::Version
-  version: 0.10.rc6
+  version: 0.10.rc7
 platform: ruby
 authors:
 - Arnaud Lachaume
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2020-05-08 00:00:00.000000000 Z
+date: 2020-05-12 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport