cloudtasker 0.7.0 → 0.9.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 92abbab630f4e60c7ae8a6b2e249a5e19d49ad9730788381790463cfda136aea
-  data.tar.gz: c71fcf022efd34d3bba1e11341583b18581d1ae93bcd128b769b2e871e5e76f6
+  metadata.gz: dc46ef6ccfab1437c91a90c3cf3e4ee7b401fe5b453d8e61c7d8af09607f6695
+  data.tar.gz: 4a8192b395ff86ff1dba2d06d677237273cef952533e41b199256db3bad2cc0d
 SHA512:
-  metadata.gz: 4a0f52436da444c75530ceb49ac16ce4c76392383011774accb31418447b6d0b2677e1289cb5768be39bb51b2665b46f347e0496c98c9552bf8c1a82a324f6e1
-  data.tar.gz: 5597abc655cef50010bbb38850cc26a5f440404aa8e3d0b645d49fe6bf34f4222d8402e1d61d3e99cf4a0093f7ccd66f561258a3b02595de7a7774cb21cda643
+  metadata.gz: 387f5202bb42b3903cf7cede2fcc84978f8ca0bf0e61eadd73d8e8fbf2d4c29ce5bb15caa749b22386ec3e5363936afb93f9600b6fd0886734909937512303bd
+  data.tar.gz: 6b3a4a77d35d1c885e623e7e737ee9008fd16e05568e3f5f2bc8e82b9ce39feecfcaf875c4b94c6dc94ecb0a68855e651759980488b1eb633bb5ca645f254c73

data/.gitignore

@@ -12,5 +12,8 @@
 /spec/dummy/tmp/
 /tmp/
 
+# Ignore lock files (e.g. Gemfile.lock)
+*.lock
+
 # rspec failure tracking
 .rspec_status

data/.rubocop.yml

@@ -1,5 +1,10 @@
 require: rubocop-rspec
 
+AllCops:
+  Exclude:
+    - 'gemfiles/vendor/**/*'
+    - 'vendor/**/*'
+
 Metrics/ClassLength:
   Max: 150
 

data/.travis.yml

@@ -2,11 +2,11 @@
 language: ruby
 cache: bundler
 rvm:
-  - 2.3
-  - 2.4
   - 2.5.5
+services:
+  - redis-server
 before_install: gem install bundler -v 2.0.2
-before_script: rubocop
+before_script: bundle exec rubocop
 gemfile:
   - gemfiles/google_cloud_tasks_1.0.gemfile
   - gemfiles/google_cloud_tasks_1.1.gemfile

data/CHANGELOG.md

@@ -1,5 +1,46 @@
 # Changelog
 
+## [v0.9.1](https://github.com/keypup-io/cloudtasker/tree/v0.9.1) (2020-02-11)
+
+[Full Changelog](https://github.com/keypup-io/cloudtasker/compare/v0.9.0...v0.9.1)
+
+**Fixed bugs:**
+- Cloud Task: raise `Cloudtasker::MaxTaskSizeExceededError` if job payload exceeds 100 KB. This is mainly to have production parity in development when running the local processing server.
+
+## [v0.9.0](https://github.com/keypup-io/cloudtasker/tree/v0.9.0) (2020-01-23)
+
+[Full Changelog](https://github.com/keypup-io/cloudtasker/compare/v0.8.2...v0.9.0)
+
+**Fixed bugs:**
+- Cloud Task: Base64 encode task body to support UTF-8 characters (e.g. emojis).
+- Redis: Restrict to one connection (class level) to avoid too many DNS lookups
+
+**Migration**
+For Sinatra applications please update your Cloudtasker controller according to [this diff](https://github.com/keypup-io/cloudtasker/commit/311fa8f9beec91fbae012164a25b2ee6e261a2e4#diff-c2a0ea6c6e6c31c749d2e1acdc574f0f).
+
+## [v0.8.2](https://github.com/keypup-io/cloudtasker/tree/v0.8.2) (2019-12-05)
+
+[Full Changelog](https://github.com/keypup-io/cloudtasker/compare/v0.8.1...v0.8.2)
+
+**Fixed bugs:**
+- Config: do not add processor host to `Rails.application.config.hosts` if originally empty.
+
+## [v0.8.1](https://github.com/keypup-io/cloudtasker/tree/v0.8.1) (2019-12-03)
+
+[Full Changelog](https://github.com/keypup-io/cloudtasker/compare/v0.8.0...v0.8.1)
+
+**Fixed bugs:**
+- Local dev server: ensure job queue name is kept when taks is retried
+- Rails/Controller: bypass Rails munge logic to preserve nil values inside job arguments.
+
+## [v0.8.0](https://github.com/keypup-io/cloudtasker/tree/v0.8.0) (2019-11-27)
+
+[Full Changelog](https://github.com/keypup-io/cloudtasker/compare/v0.7.0...v0.8.0)
+
+## [v0.7.0](https://github.com/keypup-io/cloudtasker/tree/v0.7.0) (2019-11-25)
+
+[Full Changelog](https://github.com/keypup-io/cloudtasker/compare/v0.6.0...v0.7.0)
+
 ## [v0.6.0](https://github.com/keypup-io/cloudtasker/tree/v0.6.0) (2019-11-25)
 
 [Full Changelog](https://github.com/keypup-io/cloudtasker/compare/v0.5.0...v0.6.0)

data/README.md

@@ -1,14 +1,16 @@
+[![Build Status](https://travis-ci.org/keypup-io/cloudtasker.svg?branch=master)](https://travis-ci.org/keypup-io/cloudtasker) [![Gem Version](https://badge.fury.io/rb/cloudtasker.svg)](https://badge.fury.io/rb/cloudtasker)
+
 # Cloudtasker
 
 Background jobs for Ruby using Google Cloud Tasks.
 
 Cloudtasker provides an easy to manage interface to Google Cloud Tasks for background job processing. Workers can be defined programmatically using the Cloudtasker DSL and enqueued for processing using a simple to use API.
 
-Cloudtasker is particularly suited for serverless applications only responding to HTTP requests and where running a dedicated job processing is not an option (e.g. deploy via [Cloud Run](https://cloud.google.com/run)). All jobs enqueued in Cloud Tasks via Cloudtasker eventually gets processed by your application via HTTP requests.
+Cloudtasker is particularly suited for serverless applications only responding to HTTP requests and where running a dedicated job processing is not an option (e.g. deploy via [Cloud Run](https://cloud.google.com/run)). All jobs enqueued in Cloud Tasks via Cloudtasker eventually get processed by your application via HTTP requests.
 
 Cloudtasker also provides optional modules for running [cron jobs](docs/CRON_JOBS.md), [batch jobs](docs/BATCH_JOBS.md) and [unique jobs](docs/UNIQUE_JOBS.md).
 
-A local processing server is also available in development. This local server processes jobs in lieu of Cloud Tasks and allow you to work offline.
+A local processing server is also available in development. This local server processes jobs in lieu of Cloud Tasks and allows you to work offline.
 
 ## Summary
 
@@ -18,18 +20,21 @@ A local processing server is also available in development. This local server pr
     1. [Cloud Tasks authentication & permissions](#cloud-tasks-authentication--permissions)
     2. [Cloudtasker initializer](#cloudtasker-initializer)
 4. [Enqueuing jobs](#enqueuing-jobs)
-5. [Extensions](#extensions)
-6. [Working locally](#working-locally)
+5. [Managing worker queues](#managing-worker-queues)
+    1. [Creating queues](#creating-queues)
+    2. [Assigning queues to workers](#assigning-queues-to-workers)
+6. [Extensions](#extensions)
+7. [Working locally](#working-locally)
     1. [Option 1: Cloudtasker local server](#option-1-cloudtasker-local-server)
     2. [Option 2: Using ngrok](#option-2-using-ngrok)
-7. [Logging](#logging)
+8. [Logging](#logging)
     1. [Configuring a logger](#configuring-a-logger)
     2. [Logging context](#logging-context)
-8. [Error Handling](#error-handling)
+9. [Error Handling](#error-handling)
     1. [HTTP Error codes](#http-error-codes)
     2. [Error callbacks](#error-callbacks)
     3. [Max retries](#max-retries)
-9. [Best practices building workers](#best-practices-building-workers)
+10. [Best practices building workers](#best-practices-building-workers)
 
 ## Installation
 
@@ -104,7 +109,7 @@ Open a Rails console and enqueue some jobs
   DummyWorker.perform_async('foo')
 
   # Process job in 60 seconds
-  DummyWorker.perform_in(10, 'foo')
+  DummyWorker.perform_in(60, 'foo')
 ```
 
 Your Rails logs should display the following:
@@ -157,13 +162,31 @@ Cloudtasker.configure do |config|
   # config.secret = 'some-long-token'
 
   # 
-  # Specify the details of your Google Cloud Task queue.
+  # Specify the details of your Google Cloud Task location.
   #
   # This not required in development using the Cloudtasker local server.
   #
   config.gcp_location_id = 'us-central1' # defaults to 'us-east1'
   config.gcp_project_id = 'my-gcp-project'
-  config.gcp_queue_id = 'my-queue'
+
+  #
+  # Specify the namespace for your Cloud Task queues.
+  #
+  # The gem assumes that a least a default queue named 'my-app-default'
+  # exists in Cloud Tasks. You can create this default queue using the
+  # gcloud SDK or via the `rake cloudtasker:setup_queue` task if you use Rails.
+  #
+  # Workers can be scheduled on different queues. The name of the queue
+  # in Cloud Tasks is always assumed to be prefixed with the prefix below.
+  #
+  # E.g.
+  # Setting `cloudtasker_options queue: 'critical'` on a worker means that
+  # the worker will be pushed to 'my-app-critical' in Cloud Tasks.
+  #
+  # Specific queues can be created in Cloud Tasks using the gcloud SDK or
+  # via the `rake cloudtasker:setup_queue name=<queue_name>` task.
+  # 
+  config.gcp_queue_prefix = 'my-app'
 
   # 
   # Specify the publicly accessible host for your application
@@ -215,7 +238,7 @@ Cloudtasker.configure do |config|
 end
 ```
 
-If your queue does not exist in Cloud Tasks you should [create it using the gcloud sdk](https://cloud.google.com/tasks/docs/creating-queues). 
+If the default queue `<gcp_queue_prefix>-default` does not exist in Cloud Tasks you should [create it using the gcloud sdk](https://cloud.google.com/tasks/docs/creating-queues). 
 
 Alternatively with Rails you can simply run the following rake task if you have queue admin permissions (`cloudtasks.queues.get` and `cloudtasks.queues.create`).
 ```bash
@@ -235,10 +258,15 @@ MyWorker.perform_in(5 * 60, arg1, arg2)
 # or with Rails
 MyWorker.perform_in(5.minutes, arg1, arg2)
 
-# Worker will be processed on specific date
+# Worker will be processed on a specific date
 MyWorker.perform_at(Time.parse('2025-01-01 00:50:00Z'), arg1, arg2)
 # also with Rails
 MyWorker.perform_at(3.days.from_now, arg1, arg2)
+
+# With all options, including which queue to run the worker on.
+MyWorker.schedule(args: [arg1, arg2], time_at: Time.parse('2025-01-01 00:50:00Z'), queue: 'critical')
+# or
+MyWorker.schedule(args: [arg1, arg2], time_in: 5 * 60, queue: 'critical')
 ```
 
 Cloudtasker also provides a helper for re-enqueuing jobs. Re-enqueued jobs keep the same worker id. Some middlewares may rely on this to track the fact that that a job didn't actually complete (e.g. Cloustasker batch). This is optional and you can always fallback to using exception management (raise an error) to retry/re-enqueue jobs.
@@ -262,6 +290,52 @@ class FetchResourceWorker
 end
 ```
 
+## Managing worker queues
+
+Cloudtasker allows you to manage several queues and distribute workers across them based on job priority. By default jobs are pushed to the `default` queue, which is `<gcp_queue_prefix>-default` in Cloud Tasks.
+
+### Creating queues
+
+More queues can be created using the gcloud sdk or the `cloudtasker:setup_queue` rake task.
+
+E.g. Create a `critical` queue with a concurrency of 5 via the gcloud SDK
+```bash
+gcloud tasks queues create <gcp_queue_prefix>-critical --max-concurrent-dispatches=5
+```
+
+E.g. Create a `real-time` queue with a concurrency of 15 via the rake task (Rails only)
+```bash
+rake cloudtasker:setup_queue name=real-time concurrency=15
+```
+
+When running the Cloudtasker local processing server, you can specify the concurrency for each queue using:
+```bash
+cloudtasker -q critical,5 -q important,4 -q default,3
+```
+
+### Assigning queues to workers
+
+Queues can be assigned to workers via the `cloudtasker_options` directive on the worker class:
+
+```ruby
+# app/workers/critical_worker.rb
+
+class CriticalWorker
+  include Cloudtasker::Worker
+
+  cloudtasker_options queue: :critical
+
+  def perform(some_arg)
+    logger.info("This is a critical job run with arg=#{some_arg}.")
+  end
+end
+```
+
+Queues can also be assigned at runtime when scheduling a job:
+```ruby
+CriticalWorker.schedule(args: [1], queue: :important)
+```
+
 ## Extensions
 Cloudtasker comes with three optional features:
 - Cron Jobs [[docs](docs/CRON_JOBS.md)]: Run jobs at fixed intervals.
@@ -277,7 +351,7 @@ When working locally on your application it is usually not possible to have a pu
 ### Option 1: Cloudtasker local server
 The Cloudtasker local server is a ruby daemon that looks for jobs pushed to Redis and sends them to your application via HTTP POST requests. The server mimics the way Google Cloud Tasks works, but locally!
 
-You can configure your applicatiion to use the Cloudtasker local server using the following initializer:
+You can configure your application to use the Cloudtasker local server using the following initializer:
 ```ruby
 # config/initializers/cloudtasker.rb
 
@@ -299,15 +373,20 @@ bundle exec cloudtasker
 You can as well define a Procfile to manage the cloudtasker process via foreman. Then use `foreman start` to launch both your Rails server and the Cloudtasker local server.
 ```yaml
 # Procfile
-web: rails s
-worker: cloudtasker
+web: bundle exec rails s
+worker: bundle exec cloudtasker
+```
+
+Note that the local development server runs with `5` concurrent threads by default. You can tune the number of threads per queue by running `cloudtasker` the following options:
+```bash
+bundle exec cloudtasker -q critical,5 -q important,4 -q default,3
 ```
 
 ### Option 2: Using ngrok
 
 Want to test your application end to end with Google Cloud Task? Then [ngrok](https://ngrok.io) is the way to go.
 
-First start your ngrok tunnel and take note of the :
+First start your ngrok tunnel:
 ```bash
 ngrok http 3000
 ```
@@ -318,9 +397,9 @@ Take note of your ngrok domain and configure Cloudtasker to use Google Cloud Tas
 
 Cloudtasker.configure do |config|
   # Specify your Google Cloud Task queue configuration
-  # config.gcp_location_id = 'us-central1'
-  # config.gcp_project_id = 'my-gcp-project'
-  # config.gcp_queue_id = 'my-queue'
+  config.gcp_location_id = 'us-central1'
+  config.gcp_project_id = 'my-gcp-project'
+  config.gcp_queue_prefix = 'my-app'
 
   # Use your ngrok domain as the processor host
   config.processor_host = 'https://your-tunnel-id.ngrok.io'
@@ -332,7 +411,7 @@ end
 
 Finally start Rails to accept jobs from Google Cloud Tasks
 ```bash
-rails s
+bundle exec rails s
 ```
 
 ## Logging
@@ -341,7 +420,7 @@ There are several options available to configure logging and logging context.
 ### Configuring a logger
 Cloudtasker uses `Rails.logger` if Rails is available and falls back on a plain ruby logger `Logger.new(STDOUT)` if not.
 
-It is also possible to configure your own logger. For example you can setup Cloudtasker with [semantic_logger](http://rocketjob.github.io/semantic_logger) by doing the following your initializer:
+It is also possible to configure your own logger. For example you can setup Cloudtasker with [semantic_logger](http://rocketjob.github.io/semantic_logger) by doing the following in your initializer:
 ```ruby
 # config/initializers/cloudtasker.rb
 
@@ -387,7 +466,7 @@ Cloudtasker::WorkerLogger.log_context_processor = lambda { |worker|
 }
 ```
 
-You could also decide to log all available context (including arguments passed to perform) for specific workers only:
+You could also decide to log all available context - including arguments passed to `perform` - for specific workers only:
 ```ruby
 # app/workers/full_context_worker.rb
 
@@ -402,7 +481,7 @@ class FullContextWorker
 end
 ```
 
-See the [Cloudtasker::Worker class](blob/master/lib/cloudtasker/worker.rb) for more information on attributes available to be logged in your `log_context_processor` proc.
+See the [Cloudtasker::Worker class](lib/cloudtasker/worker.rb) for more information on attributes available to be logged in your `log_context_processor` proc.
 
 ## Error Handling
 
@@ -534,7 +613,7 @@ MyWorker.new.perform({ 'foo' => 'bar', 'baz' => { 'key' => 'value' } })
 ```
 
 ### Be careful with default arguments
-Default arguments passed to the `perform` method are not actually considered as job arguments. Default arguments will therefore be ignored in contextual logging and by extensions relying on arguments such as the `unique-job` extension.
+Default arguments passed to the `perform` method are not actually considered as job arguments. Default arguments will therefore be ignored in contextual logging and by extensions relying on arguments such as the [unique job](docs/UNIQUE_JOBS.md) extension.
 
 Consider the following worker:
 ```ruby
@@ -556,7 +635,9 @@ If you enqueue this worker by omitting the second argument `MyWorker.perform_asy
 - The `time_at` argument will be ignored by the `unique-job` extension, meaning that job uniqueness will be only based on the `user_id` argument.
 
 ### Handling big job payloads
-Keep in mind that jobs are pushed to Google Cloud Tasks via API and then delivered to your application via API as well. Therefore any excessive job payload will slow down the enqueuing of jobs and create additional processing when receiving the job.
+Google Cloud Tasks enforces a limit of 100 KB for job payloads. Taking into accounts Cloudtasker authentication headers and meta information this leave ~85 KB of free space for JSONified job arguments.
+
+Any excessive job payload (> 100 KB) will raise a `Cloudtasker::MaxTaskSizeExceededError`, both in production and development mode.
 
 If you feel that a job payload is going to get big, prefer to store the payload using a datastore (e.g. Redis) and pass a reference to the job to retrieve the payload inside your job `perform` method.
 
@@ -585,6 +666,45 @@ Rails.cache.write(payload_id, data)
 BigPayloadWorker.perform_async(payload_id)
 ```
 
+### Sizing the concurrency of your queues
+
+When defining the max concurrency of your queues (`max_concurrent_dispatches` in Cloud Tasks) you must keep in mind the maximum number of threads that your application provides. Otherwise your application threads may eventually get exhausted and your users will experience outages if all your web threads are busy running jobs.
+
+#### With server based applications
+
+Let's consider an application deployed in production with 3 instances, each having `RAILS_MAX_THREADS` set to `20`. This gives us a total of `60` threads available.
+
+Now let's say that we distribute jobs across two queues: `default` and `critical`. We can set the concurrency of each queue depending on the profile of the application:
+
+E.g. 1: The application serves requests from web users and runs backgrounds jobs in a balanced way
+```
+concurrency for default queue: 20
+concurrency for critical queue: 10
+
+Total threads consumed by jobs at most: 30
+Total threads always available to web users at worst: 30
+```
+
+E.g. 2: The application is a micro-service API heavily focused on running jobs (e.g. data processing)
+```
+concurrency for default queue: 35
+concurrency for critical queue: 15
+
+Total threads consumed by jobs at most: 50
+Total threads always available to API clients at worst: 10
+```
+
+Also always ensure that your total number of threads does not exceed the available number of database connections (if you use any).
+
+#### With serverless applications
+
+In a serverless context your application will be scaled up/down based on traffic. When we say 'traffic' this includes requests from Cloud Tasks to run jobs.
+
+Because your application is auto-scaled - and assuming you haven't set a maximum - your job processing capacity if theoretically unlimited. The main limiting factor in a serverless context becomes external constraints such as the number of database connections available.
+
+To size the concurrency of your queues you should therefore take the most limiting factor - which is often the database connection pool size of relational databases - and use the calculations of the previous section with this limiting factor as the capping parameter instead of threads.
+
+
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.

data/_config.yml

@@ -0,0 +1 @@
+theme: jekyll-theme-slate

data/app/controllers/cloudtasker/worker_controller.rb

@@ -16,11 +16,6 @@ module Cloudtasker
     # Run a worker from a Cloud Task payload
     #
     def run
-      # Build payload
-      payload = request.params
-                       .slice(:worker, :job_id, :job_args, :job_meta)
-                       .merge(job_retries: job_retries)
-
       # Process payload
       WorkerHandler.execute_from_payload!(payload)
       head :no_content
@@ -39,6 +34,27 @@ module Cloudtasker
 
     private
 
+    #
+    # Parse the request body and return the actual job
+    # payload.
+    #
+    # @return [Hash] The job payload
+    #
+    def payload
+      @payload ||= begin
+        # Get raw body
+        content = request.body.read
+
+        # Decode content if the body is Base64 encoded
+        if request.headers[Cloudtasker::Config::ENCODING_HEADER].to_s.downcase == 'base64'
+          content = Base64.decode64(content)
+        end
+
+        # Return content parsed as JSON and add job retries count
+        JSON.parse(content).merge(job_retries: job_retries)
+      end
+    end
+
     #
     # Extract the number of times this task failed at runtime.
     #

data/cloudtasker.gemspec

@@ -10,8 +10,8 @@ Gem::Specification.new do |spec|
   spec.authors       = ['Arnaud Lachaume']
   spec.email         = ['arnaud.lachaume@keypup.io']
 
-  spec.summary       = 'Background jobs for Ruby using Google Cloud Tasks (alpha)'
-  spec.description   = 'Background jobs for Ruby using Google Cloud Tasks (alpha)'
+  spec.summary       = 'Background jobs for Ruby using Google Cloud Tasks (beta)'
+  spec.description   = 'Background jobs for Ruby using Google Cloud Tasks (beta)'
   spec.homepage      = 'https://github.com/keypup-io/cloudtasker'
   spec.license       = 'MIT'